Mapping the cloudiest place on earth

Previous section: El Chocó

Making Mosaics of the Chocó, or not…

I started testing out Sentinel-2 mosaicing scripts of the area in Google Earth Engine. As a free satellite program run by the European Space Agency (ESA), Sentinel-2 has a great resolution (~30 meters per pixel) for landscape mapping, is well documented, and is easy to script with in Google Earth Engine (GEE), Google’s platform / database for scientific data analysis. I intended at first here to write a walkthrough of some bespoke, technical script that would be able to force its way through the clouds in a way nobody else had done. I intended that.

Early in researching, I discovered there was already a bespoke, technical product that would do exactly this, the Sentinel-2 Cloud Probability dataset, developed by a ESA/Copernicus team.

It uses a machine-learning algorithm to derive a probability value for each pixel of every image in the Sentinel-2 collection, and it captures not only clouds but their shadows too (which ideally we would want to filter out too).

My pride had humiliated me. I just needed to write a short script taking this pre-made dataset, apply it to the Chocó and I’d be done. Even the Earth Engine Snippet example code shows off an image in Colombia, right next to where I was focusing:

Curse you Earth Engine developers…

Even worse, there is yet another product, the Cloud Score+ dataset (developed by Google themselves), which gives each pixel an ‘occlusion’ (or ‘qa’) quality, and it is built right into a Sentinel-2 database. You can filter by occlusion directly in the image collection.

I wanted to see if these two products would make things easier for me. I picked roughly around the area of Lloró, a town with the dubious honor of being one of the rainiest on earth –even at its sunniest, Lloró is overcast 3 out of every 4 days. Despite this, Lloró (and much of the Chocó) struggles with chronic clean drinking water shortages.

Some quick testing showed that both produced similar results, but neither was a knock-out success. The default Earth Engine code with the Cloud Score+ used one month of imagery at a 60% clear threshold, and it made me think I would have work to do after all,

Even with a full year of data it struggles:

There is a noticeable problem here. A pixel is not binary clear/cloudy. There is a range, and a broad filter will let in many pixels that are hazy alongside pixels that are totally clear. This gives the image an unpleasant mottled look. Alternatively, a very strict filter may not let any pixels through in some areas, creating growing data holes until there is barely an image at all.

At a threshold of about 60 – 80% clear, I can live with a foggy mosaic I can tweak in Photoshop later, but these data holes presented an intriguing question. Being able to make a cloud-free mosaic relies on there being cloud-free days in the first place – is it possible there are places in the Chocó where it is never clear?

Well, for our purposes yes, but in reality not quite. Maybe…

If we calculate the Cloud Score+ for the entire region, we notice the quality of pixels dips in consistent places inside the Chocó:

I noticed a large data hole around Cerro Torrá, an isolated Andean peak south of Lloró.

After scanning through images with the Copernicus browser I noticed that storms often rolled in from the west, and then would linger on its slopes and never dissipate before another storm came in, even while its peak was often cloud-free.

I kept thinking though, it seems unlikely that there are truly never times where the sky is clear. I also would often look through photos of the Chocó and find blue skies, and if you have been there yourself, it’s not actually overcast all of the time. There is even a photo of Cerro Torrá itself with clear skies visible over most of the mountain and surrounding area:

We have to remember that Sentinel-2 is not actually taking an image of every moment, or of every day. Sentinel-2 is taking an image at approximately 10-11 a.m. local time, about every 3-5 days. So either Sentinel-2, over its multiple years of data collection over this area, has just phenomenally bad timing, or cloud cover occurs consistently at certain times in the day (*and* Sentinel-2 has bad timing). 

Fortunately, the Colombian government has an excellent open data portal, and with a bit of fiddling, I was able to get the sub-hourly rainfall (a decent proxy for cloud cover) for every governmental weather station in the Department of the Chocó, which I will again treat as a proxy for the entire region. You can see the data I used here.

I first filtered to a single week’s worth of observations in September, when the Chocó jet is highly active, as an ideal week to capture the most distinctly regional weather pattern. There are consistent spikes in rainfall (mm) separated roughly evenly each day:

These spikes are all within 3 hours or so, of 10 a.m. local time, about when Sentinel-2 is overhead. Rainfall is least likely in the late-night / very early morning, and then rises throughout the day, peaking in the early afternoon. 

To see if there was maybe variation throughout the year when the jet was less active, I manually combined every month’s first week of values for 2020 together:

It is nothing if not consistent, and perhaps unsurprising given the equatorial climate here would limit any large swings in weather pattern. Other research backed up that the best time to get a clear sky in the Chocó is around midnight – 5 a.m (source).

Which brings us back to Cerro Torrá, and our broader project. The entire Chocó is cloudy and rainy, and to what extent varies more by location, than by time. So even if we know there are times where the Chocó is mostly cloud-free in our worst problem areas, the satellite is not overhead then, so it might as well not exist for our purposes.

At least we know now, that we are on the wrong side of weather patterns, and that we are bound more by the climate on the ground than the algorithms of the script. It makes me feel better about throwing in my hat, and deciding I’ve squeezed out what I can and I should move onto patching the data holes.

Next section: Interpolation