Toward an effective flight control software development methodology

17 min readNov 26, 2019

If we take a closer look in the most successful approaches in software development followed by the giants and also adopted by the smaller tech startups, we can identify a framework with some common elements which are actually widely spread by platforms that describe themselves as code management platform (Bitbucket), application for DevOps lifecyle (Gitlab) or software development platform (Github), just to enumerate few of them. If we track a need of change in a software (a bug, a new feature, etc) since the description of this need until the moment its implementation is delivered in production, making an wise utilisation of such platforms and supported by a robust organizational process, we can indeed provide high quality solutions in a very fast pace. Of course the tools will not make any magic by themselves and the success of this good example of agile methodology application will rely on loads of efforts to assure the maturity of the processes.

When it comes to the development of software with very critical requirements at the level of assurance and safety as flight control software, for instance, a bug in production not only means an upset customer or a substantial loss of money but may cost human lives literally. However, the methodology and tools applicable to less critical software development (mentioned previously) can be adapted to flight control software development in order to make it even safer, more robust and eventually iterating in faster cycles.

In this article I will try to exemplify the applicability of at least three concepts of software development: issue tracking, version control and continuous integration. Each concept starts with a very basic application (ready to use in most of the tools mentioned) and gradually walks toward a wish list of features that the tools should provide to fit specific flight control software development needs. Notice that we will focus here in the software related to the flight control laws and flight dynamics models which provide input, for example, to the flight simulators and ultimately to the embedded software development that has its own and very particular methodologies and processes. Despite of this, hopefully anyone in a related software development context may find inspiration to adapt these concepts in the day-to-day reality.

Example scenario

We will illustrate the concepts using an example of a personal project called McFlight. This project is a model for simulating the flight mechanics of the F-16. The model is very simple and it includes just a single control law: a pitch rate control augmentation system. Here we will describe two needs for changing the model, but before detailing them, let us put some further colors in the narrative.

Suppose that the hypothetical development team responsible for this project has, at least, two sub-teams: one responsible for the sensors model and another one responsible for the control laws model. Of course any change in the sensors model will affect directly the control laws team and they have to be aware about the updates. The control laws models eventually will turn into the software embedded in the real aircraft and before that, it will be extensively used in simulator environments to be validated. Within this framework, two changes were required to be in the next aircraft model to be delivered:

The sensors team decided to update the alpha sensor to make it closer to the real sensor behavior, and this effectively meant adding a simple-lag filter of time constant tau = 0.1s;
The control laws team decided to update the command shaping for the pitch rate controller. Before the change, the longitudinal stick inputs provided by the sensors model was received by the control laws directly as a pitch rate input (mind that this is just a very simple example);

Issue Tracking System

We can start a long philosophical discussion about the idea behind an Issue Tracking System, but luckily many people stressed out this subject in a much better way I could do it, as Vasuman Ravichandran in this excellent article. So, using very few words, we demand a system to track the materialization of our needs throughout the implementation. In our example, we will name an entity as task to express each need. We will define a task by a set of attributes to make the tracking effective.

Figure 1. Task for adding an alpha sensor.

The Figure 1 shows the task corresponding to the need of adding a filter to the alpha sensor. We can see that the task has an unique identification in the system (SENSORS-01) for possible references (as we will see later); a title, a brief description, the engineer responsible of putting the task in the system (creator), the one who will be responsible to the implementation and the one responsible for reviewing the task once it is done. We will also have a workflow associated to the task. In other words, each task will go through a lifecycle since its creation until the implementation, having multiple states and transitions rules based on the internal process of the team. In our example, the task will be created with a status To Do and once it is assigned to implementation, the status will change to In Progress, to be finally closed with a status Done, after possible reviewing iterations. This simple workflow is shown in Figure 2. We will talk a bit more about the Review status in the Version Control System section.

Additionally we can add attachments to the task for many reasons like substantiating our need (notice that the PDF file is a document from the sensors supplier) or elements to help the validation of the implementation. In this example we included a set of aircraft maneuvers in which we may observe differences when comparing time history simulations of before and after implementation. Figure 3 shows the task for the pitch rate command shaping change.

Figure 3. Task for changing pitch rate controller command shaping.

These basic features we have just described are presented in most of the issue tracking tools available (GitHub, Bitbucket, Gitlab) with their own particularities. Again, considering the context of each team, we may have available many other fields and rules associated to the tasks. For instance, we can assign each task to a deliverable, we can estimate the time to get it done, we can allow a discussion (by comments) in the task, we can build a relation between two tasks, etc. As a matter of fact, connecting two tasks may add some robustness to the process. Imagine, for example that we could state in the system that the task CONTROLLAWS-01 should be implemented/tested only after the task SENSORS-01 implementation.

Until this point, we can say that certainly any flight control software development environment has a tool or a set of tools, via software or not, to make what I have just described. But Issue Tracking System is not just a fancy named concept. Indeed we can enumerate some points to make a flight control development environment closer to a software development one:

Wise and precise use of the software tool for issue tracking. And this means that few (or none) steps in the development process should be out of the tool. If someone has to “manually” notify someone else about a transition in the task workflow, for example, like giving a call or sending an email this should sign a warning flag that there would be room for improvement in the tool and/or the process;
Preferably one single centralized tool for all teams. This would certainly make everyone’s life easier especially in terms of communication and visibility. In our example, we could think about linking the two tasks to make sure that the task CONTROLLAWS-01 shall be tested only after SENSORS-01 implementation. However if one single tool is not an option, the communication between the two or more tools must happen automatically and transparently to the users;
Smooth and resourceful integration with other development tools like control version system and continuous integration system (we will talk a bit more about it later). This is decisive in making development cycles faster;
Measurements. Loads of structured data to support management decisions. Where are the bottlenecks? Why? Which kind of tasks are taking more time In Progress status or which ones are iterating more in reviewing cycles? Which part of the project is getting the greatest number of bug reports? Which team is overloaded? And so on;

If we are convinced and decided to use an Issue Tracking tool in our contexts, certainly the best approach would be starting by a very basic framework and then making it grow incrementally, adapting it to our necessities. For this section and the next ones, I will try to imagine what would make an excellence vision (cherry on top) of the described concept. Notice that we do not have to necessarily achieve this ideal scenario to take advantage of this concept.

Cherry on top. Highest level of automation in the task lifecycle, reflecting the internal process of the department, without meaning any further burden to the user. Imagine that there is a set of items a reviewer should care about when doing his job, the tool should be able to assure that this review checklist was filled up, by, for example, elegantly infering it based on actions that the reviewer did in other integrated systems or in the issue tracking system itself. Alternatively, it could ask the reviewer to fill the checklist. If at some point, the reviewers start filling up this checklist randomly just to get rid of this bureaucracy burden, this diverges from our ideal implementation.

Version Control System

If you ever had a folder with a content like the one shown in the Figure 4, you certainly needed a Version Control System. We may be interested to roll back some changes if it is necessary or, if we recurrently deliver this report, we may need to assure which was the content in a specific version delivered. These are just few features that a versioning control can provide.

Figure 4. Prehistoric version control attempt.

Getting back to our flying world, for instance, we may face an incident investigation such that we have to assure that the stability margins of the control law inner loop were matching the design criteria for that specific software version of that specific aircraft. Using the fancy latest version control tools we would be able to browse the change history of the flight control software with a half dozen of clicks or commands. These tools provide numerous other features and there are plenty of good documentation about them on the internet (try to search for Git, Subversion, Mercurial, Visual SourceSafe, etc). In our example here, we will talk about many Git concepts but certainly the other tools have similar features.

Since we are introduced about the importance of version control, let us keep tracking of our just created tasks. In the Sensors department, John was notified that he could start working on his task SENSORS-01. He logs into the Issue Tracking System, flags that he will start working on the task, changing its status to In Progress. He takes a careful look in the task description, he spends some time understanding the supplier specification in the document attached, he analyzes the proposed filter response and finally he feels ready to start the implementation. He browses the file repository containing the latest version of the sensors model (or the version he is supposed to change) and creates a branch of this main one. To make it simple, let us say that a branch is John’s local working copy of the model in which he will be working while doing this task. In this meanwhile, the main branch may evolve. When John finishes his work, he then merges his branch into the main one. Figure 5 shows the sketch of this branching mechanism.

Figure 5. Version control for SENSORS-01 implementation.

In order to minimize the efforts in the final merging work when the task is finished, John has to be aware of updating his branch with possible new versions of the main branch from time to time.

John finally has the model on his screen as shown in Figure 6 (this model is truly available at this GitHub project) and he quickly performs his change (Figure 7).

Figure 6. Sensors model used in this example.

Figure 7. SENSORS-01 implementation.

In order to be sure that his implementation works and it is not inserting any bugs in the model, before changing his task status to review in the Issue Tracking System, John could run a possible existent set of unit tests and further ones created to test this change specifically (we will talk a bit more about testing in the next section). Eventually John is ready to send his task to review. He commits his changes in the model to send them to the file repository server.

We said previously that it is extremely valuable to have the Issue Tracking System integrated to the Version Control System (and Continuous Integration routines, as well). One simple and very good example (present at least in the Bitbucket platform) is done by using the commit message that the implementer inserts when sending his changes to the server. Let us say John writes the following commit message:

[SENSORS-01] Inserting a simple lag-filter of 0.1s in alpha sensor path.

The fact of putting the identifier of the task in the message makes the Issue Tracking System automatically list the commits in the task SENSORS-01. When Matthew (responsible for the review) browses this task in the Issue Tracking System, he is able to check the list of commits directly. Following the agreed responsibilities of a reviewer role in the Sensors team, Matthew may check if John followed the best practices of model coding (for example, maybe John was not supposed to insert the filter constant hard coded into the model), if John created a test to assure the correctness of his implementation, if there is any change out of the scope of this task in the list of commits, etc. Matthew can make comments and return the task to In Progress status so that John can fix the implementation or he can approve and sign thumbs up to John to merge his branch in the model main branch.

In this meanwhile, Ana, from the Control Laws department, is analyzing the task CONTROLLAWS-01 whose implementation was assigned to her. She is already aware about the changes to be done, but she realizes that this task makes reference to the task SENSORS-01: once the control laws gains are tuned to the newer version of the alpha sensor, if she uses a model with this sensor already updated, the validation of the pitch rate controller behavior will be more consistent with the possible previous analysis performed by her team. In summary, it would be interesting if she could use the implementation already done by John, even before the final approval from Matthew. By the Version Control System, all she has to do is to create her branch from SENSORS-01 instead of the main branch. This also may facilitate the merge when she has eventually finished this task. The Figure 8 shows the relation between the branches.

Figure 8. Version Control System for CONTROLLAWS-01 implementation.

Ana finishes her work, submit the task for review to Eve. After careful checking, Eve gives her approval to Ana. By this time, John has already merged SENSORS-01 to the main branch. Ana then does the same for CONTROLLAWS-01.

Despite of the fact that the flight control software we are talking about here is not the final software to be embedded in the real aircraft, the requirements from the aviation certification authorities also affects the flight control laws and flight dynamics models used in the aircraft development. Most of the manufacturers and suppliers (if not all of them) are producing fly-by-wire systems nowadays following the considerations established by the standard document DO-178C/DO-178B. This document has a section fully dedicated to software configuration management process which includes version control system. Having a judicious use of a version control system since earlier stages in the aircraft development may certainly make smoother the path to the certification.

Cherry on top. Sometimes the action of sending a branch to review before being merged into the main branch is called Pull-Request (at least in the Git world). When we are changing pieces of code in text files, Git is very powerful to detect the differences between the original and changed versions of the files (and eventually is also powerful in merging these files automatically), such that a typical visual interface in the Version Control System for Pull-Request, shows for each changed piece, what was added (green background color), what was deleted (red background color) and a possibility for the user to comment on that piece of code. In the Figure 9 we can see how this is nicely done by the Bitbucket.

Figure 9. Code review interface in Bitbucket.

When we come back to our flight control software development world, we notice that we have mostly models (Xcos, Simulink, Scade, etc) instead of code in plain text and usually these files are in binary format or, when they are in text file format, the logic in the model usually is not separated from the visual information such that if we change the position of a block, for example, the changed file may be considered very different from the original file in terms of the text file content. The point here is that it would be amazing to have tools automatically providing diff features to show the differences between two models. Figure 10 shows a sketch of how it could be.

Figure 10. Pull-Request idealization for model based design.

Continuous Integration

Continuous integration (CI), continuous delivery and continuous deployment (CD is acronym for both) have been concepts very common in software development. We could say that the most valuable result that they bring (or promise to bring) is to allow a small package of changes to be delivered in shorter life cycles but still relying on robust and automated tests to assure the quality of the final product.

Think about an application for e-commerce marketplace, for example, imagine that the development team decides to put a feature for special discounts. Considering the other concepts we have discussed in the previous sections, the development team creates the related tasks to this need, implements, tests and puts it in the pipeline to deliver in the production application without necessarily wait for a huge package of changes to be deployed in an eventual next release of the application. This is actually working for many Tech Companies and you may find many successful use cases examples in the internet.

Once again, when we are talking about a very traditional industry, with heavy safety requirements and many other particularities that there are in an aircraft development, it is not an easy job to adapt these techniques. And, in fact, there is a prior question of uttermost importance that we should wonder: are the continuous integration, continuous delivery and continuous deployment somehow applicable in the flight control software development? I would say that for continuous delivery and deployment, canonically speaking, we can start a long (mostly abstract) philosophical discussion of the possible applicability in the future. So let us focus in the continuous integration.

Getting back to our initial narrative, we can assume that the Sensors and Control Laws departments have as recurrent deliverables, the sensors/control laws models in a model-based design format (Xcos, Simulink, Scade) and possibly a compiled version of these models. So, the routine for the continuous integration here could be something like:

run the unit tests;
run the integration tests;
create an automatic documentation of the model;
compile the model;
send an automatic email for the interested parties.

At a first glance, this seems to be very straightforward, being the automation the only adding value when comparing to a common delivery process. However, continuous integration strongly relies on testing automation (first two items mentioned) and if this is not that easy to do in our e-commerce marketplace example, it is miles away to be easily done for our command shape change in the pitch rate controller.

In the online marketplace example, in order to test the new feature for special discounts, we would have to think about some scenarios: simulate an user logging in, making a purchase, going to the final step for payment and then check if the calculated discount is the expected one. We could vary some parameters of the scenario, like the amount of the payment, also covering the limiting cases and eventually we could be confident enough about the tests.

In the pitch rate controller example, on the other hand, the possible scenarios are actually time-dependent flight maneuvers simulations and most of time, the pass criteria are extremely hard to be translated into an automation script. Imagine how could we check that when the stick input commands the aircraft, the rising time of the pitch angle is within an expected range with some tolerances in the boundaries of the flight envelope, no overshoots much higher than the ones predicted in the linear design, no oscillating response, no saturations on the control surfaces of the aircraft, no sudden peaks of load factor variation (in order to prevent the coffee served to the passenger to spill out of the paper cup), etc. Absolutely not easy to build from scratch, using a traditional procedural programming language, at least.

But suppose that the brave and motivated team of engineers made a thorough set of maneuvers with pass/fail criteria very well programmed and covering most of the flight envelope. One day, exceptionally, there is an urgent delivery (exceptionally…) to be done, and the integration routine is crashing because of a test in a very corner point of the flight envelope, due to a numerical precision difference when comparing the failing variable with the expected threshold value. The expert engineer takes a look and concludes that is not a problem and the team decides to ignore that test “for now” because it will not be that quick to fix the test. This will be certainly the beginning of the extinction of that test suite and the main point here is that these hardly-built automated tests will not be easy to maintain either.

Still, in the continuous integration process, we can add any kind of automated routine able to produce a deliverable associated to that package like, for example, (as mentioned above) the generation of the model documentation; or the update of the database with all the signals defined in the model and associated formal requirements, etc. Also, having a very good collaboration with the Issue Tracking System and Version Control System, the continuous integration may include simple but very useful features like automatically providing the list of the additional tasks and/or commits present in the delivered package. Sky’s the limit.

Cherry on top. As we discussed, in the flight control development environment, we still have to improve considerably the way we build and maintain our automated tests in order to extract the full potential of the continuous integration concept described here. This means that it would be very welcome any solution providing the flight control engineers with a simpler (and still robust) way to test their models.

Where to start?

Certainly we should not start by trying to put a cherry on top of the implementation of all the three concepts just described. Little by little seems to be a good approach, as usual. Any of the platforms mentioned before (Bitbucket, Github, Gitlab) have what is necessary to provide, at least, a very nice traceability control which is by itself already a great added value in our kind of development. The evolution will take off when supported by the compromise between the levels of automation and customization. However, probably the greatest challenge will always be to insert these concepts in the team’s culture organically.