As mentioned earlier, Oceanforecast 2.0 is now in production and version 0.9 has been deprecated, to be removed on 2021-07-01. Despite what we said earlier there will be no change in the XML format for 0.9, as we have extended the EOL of the backend model to the same date. (The experimental XML version in 2.0 is also going away, although later in the summer).
The Probabilityforecast product has also been deprecated and will be removed on 2021-07-01. As a replacement we have added the following new variables to complete.json in Locationforecast/2.0:
air_temperature_percentile_10 air_temperature_percentile_90 wind_speed_percentile_10 wind_speed_percentile_90
We have also improved the handling of requests outside coverage for Nowcast and Oceanforecast (2.0). Any requests outside the model area (as defined by the shapefile which can be downloaded from the /coverage endpoint) will get a 422 error response. Any requests within the model area but which we have no data for (either due to lack of radar range or ocean forecasts for dry land) will get a GeoJSON response with an appropriate error or warning message.
Note that we have implemented a “snap” functionality in Oceanforecast so land points near the coast will return the nearest point at sea (this means the coordinates in the URL and JSON may be different). The new model areas can be seen on the maps here:
Finally, as you may know we have not supported unencrypted HTTP for several years, and all requests to port 80 have been redirected to the corresponding HTTPS URL. Yet, after several years we still get a lot of traffic from old systems which has not been updated to use HTTPS, which in some cases could be considered a privacy liability. As a consequence we have decided to drop this feature; now all requests to po 80 will be redirected to the front page of the API.
We’re always getting a steady stream of questions from API users, and after terminating Locationforecast/1.9 this has peaked significantly. Unfortunately, almost all of the questions are variations of two themes:
While the first one is easy to answer (read the documentation), it’s also the hardest to solve (people don’t read documentation). So again, if you have problems and must contact us, please make sure you a) have set an identifying User-Agent header and b) give us you IP address so we can check our logs for problems.
The other one is more complicated. While we have many products on api.met.no and several other APIs as well (e.g. Frost), most of the data we produce are really only meaningful for scientists, and quite complicated to process for people with only general programming experience. The data are stored as NetCDF files in our THREDDS archive, which is administered by another team. Us API guys don’t really know much about the data residing there and how to use them, so we can only forward such questions to the scientists or the thredds admins.
To direct your attention in the right direction, we have made a thredds landing page which contains a useful starting point in where you can find scientific data and documentation how to use them. This is now linked in the product list on api.met.no:
If you have questions regarding those data, please don’t use the weatherapi-adm address as this goes directly to the API admins. Instead send an email to email@example.com to ensure your questions go to the relevant personnel.
Some of you have already noticed this, but we have quietly launched a new beta version of Oceanforecast which now uses the same FORTI backend and JSON format as the latest Locationforecast and Nowcast. Unlike the others the new version will only offer JSON output, as the MOX XML format is excessively complicated and we can no longer find the spec(!).
Also note that the wave direction has been changed from oceanographic convention (“going to”, similar to currents) to the more common meteorological convention (“coming from”, similar to wind), as shown here:
0.9 XML: <mox:meanTotalWaveDirection uom="deg">89.7</mox:meanTotalWaveDirection> 2.0 JSON: "sea_surface_wave_from_direction": 269.7
Please take a look at the beta and start porting your existing applications shortly.
We’ll be polishing up error messages and verifying the data in the 2.0 beta until May 1st, when version 0.9 will be deprecated. At the same time we will switch so that both versions use FORTI as backend since the old Nordic-4km model is going away. This might mean some trivial differences in formatting and some values might be different due to more detailed simulations. You can get a preview of the MOX output coming to 0.9 here; note though that this link will be removed after May 1st:
We’ve made some extensions to the available listings, which can be summarized as follows:
This means the available lists can now support links with endpoints, like we’ve been using with /complete.json and /classic.xml in Locationforecast. Also we have added extra validation of legal parameter values for all products.
As promised for some time, NRK has made a dump of their placenames database so that users of the old Yr API (varsel.xml and forecast.xml) can now easily change from the old Yr API URL to both api.met.no and the correspond pages on the new Yr site. Note that the api.met.no string can be used with both locationforecast, nowcast and other products inside the coverage area.
The links are available for all countries in the world, in both English and Norwegian (bokmål and nynorsk). You can download them as zip archives of CSV files from here:
By popular requests, we have made available archives of historical alerts going back to January 2019. This means that developers can now easily simulate different alert types in their apps even when no warnings have been issued simply by turning back the clock. To use the archive, just specify the desired year and month with the “period” query parameter. See here for examples:
Also, we have removed the link to the RSS alert feed from the available list, so it now only lists the CAP files which makes more sense as both basically do the same thing.
This is the final product which has not yet been ported to FORTI. Due to changed requirements for the new Yr.no site we have decided to terminate this as a separate product, and instead add some more probabilty data to complete.json in Locationforecast. This is already in testing in-house, and will be launched publicly after Easter, at which point Probabilityforecast will be deprecated. Final EOL date is not yet determined, but sometime this summer is our best guess.
All good things must come to an end, hopefully to be replaced by something better. Here is an update of forthcoming changes to the weather APIs.
As mentioned previously, the old versions of Locationforecast (1.9), Nowcast (0.9) and Weathericon (1.1) will terminate on March 1st, 2021. Incredibly, almost 70 % of the current API traffic is still using version 1.9, This means a lot of sites and apps will stop working shortly. We recommend everyone not yet upgraded to start working on this ASAP. Please see our docs for more information:
The Yr API (forecast,xml and varsel.xml), developed by the Norwegian Broadcasting Corporation NRK is also set to expire this summer, along with the old Yr site (retro.yr.no). All Yr API users should port their applications to use Locationforecast/2.0, noting the following differences:
For more information on porting from Yr, see the developer site:
Also, NRK is working on a one-time dump of their location database, which will give you the correct coordinates and altitude to use with Locationforecast and Nowcast, as well as the URL to the forecast on the new Yr site. This is expected to be ready in a few days. Please follow the mailing list or one of the sites above to get information on where to download it.
As indicated, this product has long been deprecated, pending new functionality on Frost for the same purpose. Unfortunately this has not yet materialized, and now the production chain for ExtremesWWC is no longer supported. As a consequence we will turn it off tomorrow, February 5th 2021. Since this product only covers Norway and has only a handful of users, we hope you will forgive us the short notice.
After several months in beta, Nowcast 2.0 is finally officially launched. Both this and Locationforecast have now got a definite date for termination of the old versions.
Based on popular feedback we have decided to continue the XML format for Locationforecast and Nowcast into the foreseeable future. For this purpose we have added a /classic XML variant to Nowcast, which is basically identical to the version 0.9 format. This means porting from the old version should be much faster than rewriting to use JSON.
To support the XML format in the future, we will extend it to include the new symbol codes as introduced in Weathericon 2.0. Please use this instead of the numerical codes which are now deprecated. Starting today the symbol elements will look like this:
<symbol id="PartlyCloud" number="3" code="partlycloudy_day"/>
You can then link this to the partlycloudy_day.png icon which can be downloaded in Weathericon 2.0.
Some future additional changes to the XML format are to be expected. Some time within the next six months we will remove the following elements from the XML format; please don’t use them in your clients:
<temperatureProbability unit="probabilitycode" value="0"/> <windProbability unit="probabilitycode" value="0"/> <symbolProbability unit="probabilitycode" value="1"/>
Now that both products have been finalized, we have decided it is time to say goodbye to the old versions, hailing from 2014 and 2016 respectively. On March 1st, 2021 the following products will no longer be available:
Please update your systems to use the new version if you haven’t already. If you keep running into 403 Forbidden errors, remember that according to the Terms of Service you must identify yourself with a unique User-Agent HTTP request header with contact information.
As some of you may have noticed, yr.no now points to the new version (formerly the mobile site) and the old layout has been moved to retro.yr.no. The latter site is expected to be turned off in the summer of 2021; at the same time the old Yr API (varsel.xml/forecast.xml) will disappear. By this time users must have converted to use Locationforecast/2.0 for continued service.
The Yr developers at NRK are working on making a downloadable dump of the existings locations database, mapping place names to lat/lon coordinates. We’ll be back with more information on this once we have something to report.
Sigcharts has gotten a new parameter
validdate which makes it possible to
see which day a plot is valid for. This is now included in the available list.
Turbulence has had some airports removed and Vigra airport added. We have also
enabled search by ICAO code instead of location name; this will become the
default search method in the future (which means the links in available will use
icao instead of
location parameters). We recommend updating your clients to
use this at your earliest convenience.
After the very successfull update to our Locationforecast service, we have now upgraded Nowcast to a similar modern spec. New features include:
While some parts are still missing (documentation, JSON Schema, radar data for the rest of Denmark) it is currently useable enough that you can familiarize yourself with the new data format and start upgrading your clients. Currently we do not have any estimate for how long the current 0.9 version will be supported.
After having been unavailable for two years due to licencing issues, we have finally resolved the situation. Unlike previously when only a small subset was available, we are now able to offer you the whole range of EUMETSAT image products under the CC BY 4.0 license.
With over 4000 different images at any one time this also means that the complete available list is very large and may result in timeouts. We therefore urge you to filter this on your specific requirements, e.g. area, type and size.
We are currently working on updating our Oceanforecast service to the new JSON standard. We have also started on a new project for improving our map services (as seen on Yr.no). During our cooperation with the University of Oslo we have gotten valuable feedback from students regarding map integrations on mobile app development, but we are also very interested to hear from web developers what kind of maps you would like to see and which protocols and formats you would like to see supported.
If you have any feedback, please send us an email at firstname.lastname@example.org with the subject “Map feature requests”. We will later collate this and use for planning new map products which hopefully should be available next year.
For the last couple of years we have been working on a new forecast backend (FORTI) to replace the decade-old, PostgreSQL-based store (WDB). While Locationforecast/1.9 has been running from FORTI for some months now in “compatibility mode”, we have finally finalized the new JSON format which will become standard for all weather forecasts soon.
Version 2.0 comes in three different flavours:
This service endpoint will return a response with all available forecast parameters. As we make new forecast parameters available, they will be added to the responses for this service endpoint (we expect to include probability percentile data fairly soon). Also, some geographical areas will have more forecast parameters available than others.
This service endpoint will return a response with only a core set of forecast parameters. All forecast parameters in this endpoint will be available for every location. We will add new parameters to this endpoint as well, but much more rarely, and the size will not increase much. If you feel there is some useful parameters missing here, please let us know.
This service endpoint exists only for backwards compatibility purposes. The parameters in this endpoint and the XML format provided are identical with /locationforecast/1.9, but with more time intervals. Use this if you have an existing application and want to do minimal work to update it. We don’t expect adding any more parameters (XML tags) to this format in the future.
We have also introduced an improved topography for the entire world, which should give more accurate temperature forecasts in some areas. This topography will be used if you do not specify altitude in your request. Note that the improved topography is still relatively coarse. We therefore recommend that you specify your own altitude in the request if you have access to your own topography data.
With great power comes great responsibility. Regardless if you’re Voltaire, Churchill or Spider-Man, all users must now adhere to the Terms of Service (more below) more stringently. In particular you must pay more attention to the following rules:
You must identify yourself with contact information in the User-Agent request header. Whereas requests without any such header or just a generic string (e.g. “okhttp/3.12.4”) until now has been throttled, new products will return a 403 Forbidden instead.
Truncate all geographical coordinates to max 4 decimals. There is not point asking for forecasts down to nanometer precision, and makes it impossible to cache data. While we previously have truncated the requests in the cache, this had to be done on a per-product basis. For new products we will instead implement this as a site-wide rule, where requests with 5+ decimals will return a 403 Forbidden.
/weatherapi/locationforecast/2.0/?lat=... have been deprecated. While they will continue working for a limited time, you should always use one of the endpoints above, preferably /compact.
Locationforecast 2.0 will be officially launched on Wednesday 17 June 2020. At the same time version 1.9 will be deprecated. A final termination date has not been set yet, but we plan on turning it off before xmas.
The already deprecated LocationforecastLTS will terminate on 1 Sept 2020. This gives you almost three months to port your applications to Locationforecast 2.0, if you havent already started.
UVforecast has been deprecated as UV data is now included in Locationforecast 2.0 JSON, and will also be terminated on 17 June.
We also expect to launch a version 2.0 of Nowcast in about a month, using FORTI and the same JSON format as Locationforecast.
This is mainly a rewrite and clarification of the existing rules. While there is nothing radically new under the sun, the old TOS was confusing and badly organized, so many users missed vital points like User-Agent identification and started asking why they were being throttled. Hopefully this version should be clearer and more detailed.
The new TOS is still in draft state, but if you have any questions please let us know before we finalize it later this month.
There have been some minor changes to some specialized products:
The “airquality” profile has been removed
New names of counties and municipalities according to the 2020 reform. E.g. “Buskerud/Røyken/Midtbygda/Røyken” has been changed to “Viken/Asker/Midtbygda/Røyken”.
Hope you are all doing ok in these difficult times. Here at MET Norway it’s business as usual. Most of us are working from home, but since the API team already did this for half the week there’s not much difference. Here’s a summary of the changes so far this year:
The following products have been deprecated and will be removed on
15 May 1 June 17 June 2020:
The following have also been deprecated, probable EOL times as indicated:
Version 2.0 is nearing completion, the only major feature missing is documentation. We have decided to split the output into three different endpoints:
Also, the unit for temperature has been changed in JSON from “C” to “celsius” to conform with CF standards.
We are also happy to launch a new set of corresponding weather icons. Unlike the former numerical codes in 1.9, the 2.0 “symbol_code” variable is based on a mnemonic code and correspond exactly to the icon filename (minus extension). All icons are available in PNG, SVG and PDF formats, and can be downloaded as a gzipped TAR archive:
In addition you can also get a table matching the new and old symbol codes, as well as explanatory texts in English and several variations of Norwegian:
Radar have been upgraded to version 2.0, with new parameters, new areas, new sizes and a new projection. GIF animations will still be producted for the time being.
The new version 2.0 delivers both SIGMETS and AIRMETS, but now also includes wind shear warnings. You can now ask for each type separately, but by default you will get all in the same response. Also the long-standing bug where warnings would be included twice have finally been fixed.
This was previously part of Upperwindweather, but has now been launched as a separate product.
As mentioned last year we have deprecated the LocationforecastLTS product, which will be retired some time after we have launched the new Locationforecast 2.0. In the mean time, the background model for the Nordic area (MEPS) have been replaced with a new, improved model (CMEPS) which is run every hour.
Unfortunately, some little used parameters are no longer present in the new model, which means that some slight changes have been made to locationforecastlts:
These changes are only be visible for the Nordic area (Norway, Sweden, Denmark and Finland). The rest of the world (which uses the EC model) remains unchanged.
Some of you may have noticed that about half of the municipalities in Norway got new codes (“kommunenummer”) this year. This was implemented in MetAlerts this March. As you cannot search on municipalites yet (only counties) rhis does not affect the API interface, but if you are extracting kommunenr from the CAP files you should search for both old and new numbers.
In March we changed the file storage backend for non-dynamic products. This should result in faster response times and greatly reduce the occurence of delivering stale data.
We hope you all have been good developers this year, in which case you can look forward to some new improved products next month (for those who have been bad we will also be taking some away).
On December 3rd we launched the new backend for locationforecast/1.9. In connection with that some of our users have noted that the topography for Norway is not as accurate as before. This is because we use a terrain model with a slightly coarser resolution. This does not affect the quality of our temperature forecasts. However, due to requests from our users we will soon return to a higher resolution topography for Norway.
Next year we also plan to introduce a higher resolution topography for our global forecasts. This will make the forecast more accurate when not using the recommended “msl” parameter to indicate elevation, which is hard to implement e.g. in mobile apps.
We are also working hard on finalizing the JSON format in 2.0, as well as improving the weathericon symbols so that you won’t have to call sunrise to know if to use a night or day version.
Due to popular demand, we have decided to continue GIF animation in radar/2.0 which will launch on Feb 3rd. Everything else will be different though, included sizes (only one), areas (previously called radarsites), types (more options) and formats (all static images are now in PNG).
Next year the sigcharts in upperwindweather will be moved to a new product, possibly with some new charts to follow. The upperwindweather product will then be deprecated, and eventually removed from the public api.
We will also be launching a new version 2.0 of icemap with more and better maps of the Arctic. The old version 1.5 will continue running until March.
We have also decided to retire the little-used lightning product on api.met.no. Since you now can download lightning data more or less in real time from Frost, there is little use for 5 min old data from the API. The data will continue to be delivered in UALF format as before. You can test out the new product here (remember to register for a developer key).
Finally the radar/1.5 and uvforecast products will be end-of-lifed on May 15th, to be replaced by the improved radar/2.0 and locationforecast/2.0 respectively.
We will shortly be making some changes to the existing versions of Textforecast and Locationforecast.
The seahigh and seabank products will be replaced with a new, auto-generated product (named sea) which will be available on December 3rd (possibly earlier). The old forecasts will continue production until February 1st, after which requests will be returned the new sea forecast instead.
We will shortly be changing the backend for the locationforecast to a new, cloud-based service. For the immediate future there will be very little change for the users except for higher performance. There are some minimal changes to the version 1.9 output, in short:
Only 1- and 6-hour forecasts will be supported, the 2- and 3-hour intervals will be removed.
We have added precipitation minvalues and maxvalues for the whole forecast for the Nordic and Arctic regions, not just the first two days:
The EPS, LOCAL and EC* models in the meta header are no longer relevant.
We now how only one model (met_public_forecast), with a single
The global forecast (outside the Nordic and Arctic regions) will be shortened by 6 hours. As a compensation it will update every 6 hours, instead of every 12 hours as previously.
If you experience any problems due to these changes, we suggest temporarily switching over to LocationforecastLTS until you can make the necessary changes.
As Locationforecast and LocationforecastLTS currently runs on the same old
backend and is virtually similar, we have decided to end-of-life this product
as it is impossible to guarantee the same output on the new backend. This
means LocationforecastLTS will be officially deprecated once the new model
is in production, but will not be terminated until the end of February 2020
at the earliest (which should be plenty of time to change your
implementation). Since the new backend is much faster and has a more correct
forecast, we suggest changing over to I
Some of the planned changes for the next 6 months include:
In the long term we will be launching a new 2.0 version, which features:
At the same time we will be deprecating the Probabilityforecast and UVforecast products since the data will now be available in Locationforecast instead.
UVforecast: These are produced by external parties, and we experienced distribution problems last summer. Since UV data will be available in locationforecast/2.0, the UVforecast product will be terminated after Easter
ExtremesWWC: This product will be replaced by a similar feature on our observations and climate API frost.met.no.
We do not have exact dates for the above changes, but will return with updates once we have more specific information.
Making an API is a thankless task. Most users aren’t aware they are using it, and you rarely hear from the developers unless something isn’t working. While our weather site Yr is hugely popular, our API doesn’t get much publicity. So when we finally get some exposure, it’s sad to find only negative publicity, and misguided criticism at that. But before addressing the issues, let’s watch the presentation (starting about 8:44 into the video).
To start with, met.no isn’t a company, it’s the Norwegian Meteorological Institute – a state-owned, non-profit government organisation which has as one of its missions to give free (both as in beer and in freedom) weather data to the whole world. So far we’ve managed to do this without any developer registration or other conditions than nicely asking not to overload our services. As a consequence we have no clue about most of the thousands of apps and sites using our services, and no way of contacting them unless they subscribe to our mailing list or read the API front page.
From the interview one would get the impression we are making apps ourself using unsecure communications. Such is not the case – the only app we have some influence over is the Yr weather app, which doesn’t speak with our API at all. So already the (admittedly valid) criticism is targeted at the wrong party.
Disregarding that “it” in this case is an app we have no control over, we have actually supported (and recommended) HTTPS for many years (although in the old API this wasn’t default because of compatibility issues). Since the rewrite and launch of WeatherAPI v.3 (which came out in beta in 2016 and was gradually phased in on a per-user basis over the next year) we have only supported HTTPS. All unencrypted HTTP traffic is either being redirected or blocked (more on this later).
To this day we still get complaints about this, e.g. when no longer being able to use Squid as a local caching proxy, or using client libraries which don’t support SSL/TLS. Incredible as this may seem in 2019 it is still an issue, and when your application is a watering plant run by a Saia-Burgess PLC controller there is no sofware upgrade path.
About half of the requests to api.met.no are either throttled, blocked or invalid. Of the latter, most are for products which were removed years ago. We’re still getting requests for locationforecast/1.8 which was removed in 2014(!). Most developers don’t bother checking the response status codes, so lots of sites and apps go on blindingly ignoring 304, 404 and 429 errors for several years. And since many users don’t update their mobile apps until they have to, we’re getting a lot of legacy traffic, included unencrypted HTTP which we’re unable to prevent people from sending us.
As an example, let’s look at the HTTP traffic from Android (pre <5.0) apps where the developer haven’t read the TOS and uses the default “Dalvik” User-Agent header which amounts to 13% of the total unencrypted HTTP traffic. These are the amount of requests sorted by status code between 12 and 13 today (UTC):
As the table shows, the unencrypted traffic amounts to a whooping 43% of the total identifying only as Android (and an obsolete version at that). How much of this actually handles the redirect is difficult to see as we don’t track users, but we find it hard to believe that 76% percent of the HTTPS traffic comes from redirects. This means that a large portion of clients never follow the redirect.
Of those who use HTTPS, an incredible 55% of the HTTPS requests are being throttled for too much traffic or missing identification, while only 43% get a meaningful answer. The remaining requests (apart from a handful of timeouts) are either for products which been discontinued (sunrise/1.1 is still very popular) or they can’t manage to construct a valid URL. That’s a lot of developers who can’t be bothered to RTFM.
Now, this is where it gets interesting. Most of the HTTP traffic is actually for locationforecast; sunrise doesn’t count for more than about 5%. Of this, 5 out of 6 sunrise requests are actually zombie apps calling version 1.1, which was discountinued in February and returns a 404. (Incredibly we also get some for version 1.0 which was EOL in 2016!)
Still, that leaves about 1% of HTTP traffic for sunrise/2.0, which has never worked with unencrypted HTTP. Every piece of documentation, example code and tests we have published have been using HTTPS, so why are people still using HTTP? These requests are (if identified at all) coming from Android, so either there is something amiss with the Dalvik HTTP client stack, or developers are intentionally breaking security by rewriting our examples from “https:” to “http:”. As the saying goes, it’s hard to make stuff fool-proff because the fools are so smart.
Many of the requests do indeed contain geocoordinates. However, the vast majority of these originate from servers located in data centers totally remote from the geolocation in the request. Even when an app enable location services on your phone, we have no way of knowing if the coordinates are your actual GPS location or if you’re just checking the weather for your next trip to Paris. We’re still getting ~25,000 different IP addresses from all over the world downloading the weather forecast for a cornfield in Poznan, PL every 5 minutes, amounting to 3.15% of the total api.met.no traffic.
I confess this claim has us stumped, and after putting our collective heads together we still can’t figure out what they’re getting at. Sure, the MAC address is an identificator which can be used to link with personal data, but that is never transmitted to us (what the app developers are doing with your personal data you must ask them about).
Now, the only third party getting access to both your MAC address and your unencrypted traffic is either the owner of your Wifi access point/router or your phone operator. Admittedly, both these can be spoofed and your traffic intercepted by evil middlemen. However, in both cases they can already pretty much tell your geolocation even without snooping your data traffic. When a phone is connected to (in this case) a 800mW faked wifi access point and indoors it is pretty much given that the person is within a 50 m radius. In fact this is how shops track customer behaviour, logging how much time each person spends before each isle. Now, having the GPS coordinates in addition would probably give you a better resolution, but then GPS reception isn’t great inside buildings anyway.
Sure, the app developers can (and probably will) track you, and the same goes for your phone company or free Wifi provider. But that has nothing to do with HTTP encryption, and very little to do with geolocation (which they know already).
Now, you might ask why we didn’t go directly for HTTPS in the first place. At the time api.met.no was lauched in 2007, this was not usually an option; Facebook didn’t offer HTTPS until 2011, and many client libraries did not support it. While this now has changed, our main priority has always been backwards compatibility, something we’re proud of having had for 12 years now, even though individual products and versions have come and gone.
As mentioned earlier, all of the unencrypted HTTP traffic is either blocked or redirected, in the latter case to a similar HTTPS URL. Now, since we’re mirroring the original URL in the Location response header, it can be argued that we still “support HTTP” since legacy apps can still go on working if they handle the 301 properly. If they instead got a 404, the theory goes, they would see the error in their ways and change their code to use HTTPS instead.
Unfortunately, in practice that is not the case. The sites topping the list of HTTP have all been blocked for abuse and get a 403 Forbidden response. They have been blacklisted for years, but still continue to hammer our servers with 14k req/hour each with no end in sight. For these large websites there is no connection between the IP address and the geolocation received by us, so any “leak of personal data” here would be between the browser and the website if they’re not using HTTPS.
If we can conclude that policing obsolete websites is futile, then this goes double for obsolete mobile apps where it doesn’t matter what the developer does unless the user updates the app. As can be seen from the previous table, a lot of the traffic isn’t only against services which no longer exist, but are also more than two years old and run on obsolete versions of Android. There is not much hope that these apps will be updated any time soon.
Anyway, as a precaution we tried disabling the redirect to HTTPS to see what happened. Not only did we get a lot of support tickets from developers who couldn’t figure out what was wrong, it also broke the website. Whereas you previously could type e.g. “api.met.no/apis.json” in the browser location bar and get JSON back, the browser actually defaults to unencrypted HTTP which no longer works. Instead you actually have to type “https://” before the hostname. Can you think of any other sites in 2019 where this is necessary? Me neither.
In fact, all major sites we have tested (e.g. Google, GitHub, Paypal, eBay) redirect HTTP requests to corresponding HTTPS URLs, which is also recommended in all best practice documentation we have found, including the HTTP Strict Transport Security. Here is an example of GitHub doing exactly the same thing:
$ GET -Se http://api.github.com/users/octocat/orgs GET http://api.github.com/users/octocat/orgs 301 Moved Permanently Connection: close Date: Fri, 23 Aug 2019 11:08:56 GMT Location: https://api.github.com/users/octocat/orgs Content-Length: 0 Client-Date: Fri, 23 Aug 2019 11:08:57 GMT Client-Peer: 184.108.40.206:80 Client-Response-Num: 1 GET https://api.github.com/users/octocat/orgs 200 OK
So we’ve rolled back the change, and until we can find a way to differentiate zombies from actual live humans we’re keeping it that way.
So, the next time you hear someone boldly stating that
"APIs leak personal data"
you would be wise to interpret that as
"abandoned zombie apps, written by unknown parties for a defunct API, who send unsolicited, not-very-personal data to non-existent services, can be intercepted by hackers who will learn nothing they don't already know, and there is nothing the API owner can do to prevent it".
Admittedly that won’t make any headlines at a security conference. Instead,
if there’s anything to learn from this, it’s that most traffic against APIs
is actually noise, which you’ll just have to learn to deal with. But
admittedly that doesn’t make for good soundbites on