2014 Site Updates

Some links and images are no longer available in our older site updates. Additionally, some information is no longer applicable.

For the latest site updates, click here.

Today our old recon system was retired. Transmission problems making HDOBs come in too close together when once again available, combined with sonde data coming in too close together when passing through the center of a storm, made the old system fail to retrieve obs in real time and thus be too unreliable. Therefore, our new recon system for the North Atlantic, Eastern North Pacific & Central North Pacific and Western North Pacific basins has now officially replaced all of our older recon systems which no longer operate on our site.

A lot of work has been done over the past month. NOAA AXBT data is now processed in real time. The historical HDOB decoders have yet to be completely rewritten, however the parts that decode the observations to create Google Earth mapping have been rewritten. All historical recon data on our site can now once again be viewed in Google Earth.

Over the past two days a lot of work has been done on the new recon system. After a lot of testing, including additional fixes to special non-tasked mission handling for missions that span past midnight Zulu, the new recon system has now been released for beta testing. It has become the primary system available on our recon page, with our older recon system still available. While there is still a lot of work to do and new features to add, the work done to date has represented the most challenging parts of the new system. This morning the system to check the NHC recon archive as a backup was successfully integrated into the new recon system and was tested today for the three-plane research mission into Edouard. Our old system simply missed most of the data in real time, while the new system, aided by checking the NHC archive intelligently, was able to fill in data automatically in a reasonable amount of time. Given the ability to be more timely, the new system was moved to the front page.

Up next, adding AXBT data into the system. Due to the multiple plane NOAA research missions I wanted to complete the NHC recon archive check first so it could fill in the data for the missions.

Also coming soon, radar data from the NOAA hurricane hunters, when available. A link exists on our recon page to the temporary archive of radar data. Eventually it will be included in the live system.

Then it is on to rewriting some of the older HDOB decoders in order to be able to map older data. Then it is on to doing live recon in Google Maps.

Various updates have been made to the new recon system, reducing its stability. Some of the code that creates the recon files for Google Earth was rewritten. Some additional display enhancements were made in terms of the popup balloon when you click a plane icon and the mission path. The rewrites on the others parts of existing code are in part to make it easier to make way for writing the code to process live NOAA AXBT data. The ability for an administrator to add AXBT data into the system is up next, followed by a return to working on the ability for the system to fill in missing obs by checking the NHC's recon archive automatically. An additional NOAA recon feature will also be added, either when the NOAA AXBT feature is added, or perhaps a little later depending on testing, in a few weeks. As always, if you see any errors, please let me know.

The administrative feature to manually update the recon system using the NHC's recon archive was successfully tested online in the new recon system. That feature will now be brought to the live system to operate, occasionally, in real time to pick up missed obs. That will take some time to implement. Then it is on to AXBT data.

Parts of the mission identifier handling in the vortex decoder were rewritten in order to recognize corrected observations in a better way. This will be used in the system to check the NHC recon archive. Additionally, some planning took place on how AXBT data will be organized in the live system in real time. The main script that processes obs into the system has underwent a lot of updates in order to add admin options for the system to check the NHC recon archive.

Having the new system create Google mapping after all obs have been processed caused some errors that were corrected Monday night. Work continues on the system to allow an administrator to add obs from the NHC recon archive into the system. Additional options are being added that is requiring a rewrite to some of the decoders in how they handle corrected observations. If it works, an administrator would have the ability to choose to replace existing obs in the system only if the ob they are replacing it with is corrected. A lot of the decoders for each product handle corrected observations in various ways. Some of that is being more centralized. Additionally, the ability to recognize corrected observations is being improved in some decoders.

Additional updates have taken place. Each time the live system runs it now creates mapping after it has processed all obs into the system. If the system processed two obs from the same mission at the same time it would previously recreate the map twice, which is a waste of resources. Google Earth plugin mapping has been updated slightly, with more updates to come in the next week or two. In testing offline, the ability for an administrator to download and process obs from the NHC recon archive has had its first successful tests this morning. More features need to be added as there are some options that have not yet been completely coded. After that, a live version, that checks the NHC archive automatically at times, will be implemented in the new recon system. This will mean that data will automatically be filled in if it is missed in real time when checking individual product files up to every minute.

Considerable testing and updates took place in real time on Friday night on the new section of code to handle non-tasked missions that span across midnight Zulu. Testing took place on the actual mission and then obs were added and removed to simulate various scenarios. The system now appears to work using the new handling.

Tonight the NASA Global Hawk mission off Africa will be the first test of automatic non-tasked mission data handling of missions that last past midnight Zulu. Work continues on checking the NHC's recon archive as a secondary resource. The first step, the current task, is making the system run on demand for specific folders as an administrative option.

Additional issues were fixed this morning. Data came in for an ob from NOAA that was posted a day late and then obs from the Air Force started coming in from the year 2095. I have updated the code to reject such instances. For now, in real time, both the operational and new recon systems are rewriting HDOBs coming in with the year 2095 with the current date. I will adjust other ob types with the wrong date later tonight if needed.

Additional errors were corrected for the first East Pacific live mission in the new system. The basin network link will have to be downloaded again for all basins as the file that is called from the main file you download has changed. This morning I added what should from now on automatically handle all non-tasked missions that span multiple days. Mission data will automatically be moved from the previous day to the current day. It has yet to be tested in real time.

As expected, there has been a lot of various errors in the new recon system. I have fixed the issues as they have come up and continue to add new features. I am currently working on handling non-tasked missions better when spanning multiple days. I am also working on the Google Earth plugin mapping display. After that I will continue working on the part of the system that checks the NHC's recon archive as a secondary resource occasionally, filling in missing data automatically. Then it will likely be on to product and aircraft pages, followed by displaying AXBT data in the archive and then the live system. Then there will be further decoder rewrites and other various things before mapping finally returns for older missions. Later, perhaps late in the year or early next year, the live recon system should come to Google Maps.

After all of that I will return to updating the model system next year.

On August 26, the new recon system experimentally went online. The new system will likely continue to run live until it finally becomes our operational recon system next year. Our current, more stable recon system, will continue to run live until next year. If you notice any problems with the new system, or have any comments, please contact us. Updates will continue over the next many months on the system.

The following applies to the new recon system. I am currently working on the central part of the new recon system, rewriting the code that will actually check for recon data. The new system will use more intelligent means of checking for data, only checking for data when it thinks it is needed in order to be less demanding, including also checking the NHC's recon archive to fill in missing data automatically. I have rewritten the system for the public to add data from the NHC's recon archive and will adapt that code to be usable by the system that will check the NHC's recon archive occasionally. All of this will be managed by the system I am writing now. I have already planned out how much if it will work, including the new files needed to run it. These files will also eventually give more information on the front page, and live data pages by product and aircraft, that will give more information about the recon currently being performed.

I hope to have the new system online for the NASA Global Hawk missions which could start as early as the end of this month. Again, while track data for the Global Hawk will not be included in the new system, sonde data will be available in Google Earth as long as it is still transmitted across using NOAA's UZNT13 KWBC header.

Other features of the new recon system will come later in the year, into next year. I have yet to rewrite the older HDOB decoders, so Google Earth mapping of data from 2006 and earlier will not be available for awhile still. Google Earth mapping may be added back for data after that within the next month or two. Other new features coming to the recon system include live NOAA AXBT data and possibly delayed Air Force AXBT data. And of course the main feature still to come, likely sometime next year, live recon in Google Maps.

Work continues on getting enough of the new recon system online to operate by the end of this month for NASA's GlobalHawk missions.

I am working on getting enough of the new recon system online before the end of August in order to allow the system to process GlobalHawk dropsondes in real time for this year's HS3 mission. The new system will likely have little to no new features other than that until 2015. In fact, it will have less features than the operational system, with no Google Maps mapping, either in the live system or in the manual decoder. The live system will have Google Earth mapping. The primary system will remain the main system until sometime in 2015. Both systems would operate in real time during the HS3 mission. I'm not sure if the new system would continue to operate in real time after the month long HS3 mission.

I'm still not sure when I will process older recon data to create Google Earth files again. I have yet to work on the system that will eventually be constructed to do that.

The operational recon system is now fully operational for the hurricane season, checking vortex messages, RECCO messages and Atlantic Air Force backup files (KBIX) for data. When in off season mode, it did not check these files. The site is now fully operational for hurricane season.

I have once again revisited coding some of the recon system. So far this year I have worked exclusively on the new model system which will still take many months to complete. However, since that is definitely going to take a lot longer, the versions of the model system that are currently operating still work and testing of the new model system will take a lot of time once finished, I have decided to again work on both systems at the same time. I would really like to have the new recon system cover dropsondes from the NASA GlobalHawk in real time. Hopefully I will be able to get enough of the system working in time for that. The older recon system will remain operational until 2015 as the new system will have limited functionality until fully implemented later in 2015. Google Maps and Google Earth plugin mapping will not be available until 2015. Google Earth mapping would be made available in real time once the new recon system is operating in real time. It would be very prone to errors and outages.

It continues to make sense to work on both the model and recon systems simultaneously. They share some of the same coding and style. There is a planned addition of a new feature for the model system that will utilize something developed a few years ago for the Google Maps recon system that will still not be ready until 2015. The new feature for the model system, which may allow users to compare model data from prior runs, will need to utilize compressed json files. (This new feature may not ultimately pan out as it is in the very early development stage.) Using compressed json files is how the Google Maps recon system will eventually operate, although that technology will likely be implemented in the model system first if the new feature is successfully implemented in the new model system. However, it would be a good idea to study how it will be implemented in both systems as the mechanism for retrieving, uncompressing and making the data available for use will involve much of the same coding.

There remains a massive amount of work to do on both systems. I will provide updates on the progress of the new systems, most especially the new recon system, throughout the hurricane season. It is likely that by the end of 2014 the new and existing operational model and recon systems will run side by side, with the new systems available for testing and feedback and the older, operational, systems available for stability and the existing features that the new systems have yet to have implemented.

All the links on the site were checked and updated as needed. Our Google Earth overlay file was also checked for broken overlays and updated.

Due to coding backup methods to handle complexities with the "tcweb" folder in the past, its removal did not require too many changes in our model system. As a result, our model system is now back online, although it has moved temporarily. Our system already had the ability to force checking of the "btk" folder and with a little coding it was able to do so without first requiring a check of the "tcweb" folder. As for renumbering, we will have to wait until a storm is renumbered to determine if that is handled well. We previously had a method in place to attempt to renumber based on storm details rather than rely only on renumbering files in the "tcweb" folder. The newer model system currently being worked on offline should have better handling of renumbering. The new model system will be installed later this year. The older, operational system, will continue to update at its temporary location until the new system is installed where the old one used to be. Temporary redirects are in place.

Our model and best track system will no longer operate until changes are made to our system to reflect changes made to the availability of certain raw best track files that our site has depended on to manage data. I do not currently have a timetable on when our model system will once again be operational. A rewrite of the model system has been underway all this year. I do not yet know if I will patch the old operational system, an older version of the new version that was not yet available on our site or rush the latest version of the system, with further updates and new features added later. The older data has been moved so that it can remain accessible. It will remain until the new system is about done, with the further updates and new features added, likely late in the year.

The problem is due to the "tcweb" folder no longer being available. Our site will no longer have access to the files that helped to determine what an invest was upgraded to for example. Our site will now need to check the "btk" folder and then determine independently, in real time, what an invest has been upgraded to. We do have features of our system that do both of those things, but as a backup. It will probably take some time to code it as the primary method and remove all sections of code that checked the "tcweb" folder, and eventually rewrite all of the sections of code, something I planned to do anyway.

As for new features in the model system, those are still in the planning stages and due to the required updates to be made, along with other rewrites of parts of the model system I am doing, it will be awhile before they are possibly completed.

I am very behind on the recon system. It is likely to be 2015 before it is done. It may be late in the year before I can once again get back to working on it.