Blog
Debugging Amazon S3, Lambda, Timeouts, ContentOperationNotPermitted, wkhtmltopdf, and Headless Chrome
Intro
You ever run into a bug that is so perplexing you don’t even know where to begin? I’d like to share a story that happened this week. As you may know, Api2Pdf runs on AWS Lambda. AWS Lambda is Amazon’s serverless compute service that lets you scale to countless requests very easily.
Our two most popular API endpoints generate PDFs using wkhtmltopdf or Headless Chrome. Your choice, of course. You pass Api2Pdf some HTML, it spits back out a PDF.
However, a customer emails in this week with a strange problem. Take the simple HTML below.
| <html> | |
| <head> | |
| <link href="some-link-to-a-css-file" rel="stylesheet"> | |
| </head> | |
| <body> | |
| <p>Hello World</p> | |
| </body> | |
| </html> |
by
The most basic Hello World that included a link to a style sheet. For the sample above I redacted the actual URL to the stylesheet, but the css file was pretty large and it was hosted on Amazon S3.
The Problem
Our Headless Chrome endpoint was timing out on generating a PDF out of the HTML above. The css file must be the cause, right? Something about it must be breaking our Lambda function. This was worrisome. Our entire business is about generating PDFs and if we cannot even convert a Hello World HTML, what good are we?
Here’s the kicker – the CSS file was hosted on Amazon S3’s West Region. But if you move the exact same file to Amazon S3’s US East Region, the PDF generates just fine!
How could a single Lambda function fail when pulling a file from the West region, but succeed when pulling the file from the East? This made no sense. Surely Lambda, which is hosted on AWS, would have no problem pulling a file from another data center?
Out of sheer curiosity, I decided to see what happens if we use the wkhtmltopdf function instead of Headless Chrome. I run the tests against the East and West css files and I nearly lose my mind. Wkhtmltopdf has no problem generating a PDF from the West file, but fails on the East file!
What the f…..?!? Just to recap:
- Headless Chrome generates the PDF from the East, fails on the West.
- Wkhtmltopdf generates the PDF from the West, fails on the East.
At this point I am so confused I don’t even know where to begin.
Take a step back
What was strange about the way wkhtmltopdf failed was that it did in fact give an error. Specifically a “ContentOperationNotPermittedError”. Unlike Headless Chrome which just times out after a while. Even the way the two engines fail is different…
I google the error and come across this github post. I’m not really following or understanding the discussion as it ends up linking to another issue regarding ContentNotFoundError. This other error has more information, however. Apparently the ContentNotFoundError happens if you are linking to a resource that does not exist. I’ve seen this with other support tickets but it is usually because the css file they link to is using relative links instead of fully resolved urls. But maybe there are some relative links inside the css file?
Eureka
The breakthrough moment was when we discovered that within the CSS file there were 18 places in which relative links were being used. The relative links were trying to pull in a font at location /assets/fonts/OpenSans-Regular.woff.
In hindsight I could have just loaded the HTML file in chrome and used the Development Console to discover that it failed to load resources. Seems obvious now, but I was thrown on to the wrong trail early on and never bothered to look at the simple things.
But there was still something that remained unexplained. Even with the relative links which no doubt would cause problems, why would Headless Chrome and wkhtmltopdf succeed / fail on different regions?
Because of the relative links, they were resolving to the following:
East: https://s3.amazonaws.com/assets/fonts/OpenSans-
West: https://s3-us-west-2.amazonaws.com/assets/fonts/
Each region handles the failed asset different. East throws a Forbidden, Access Denied, while West straight cancels the request. Take a look below:
East
West
Why Amazon S3 returns two different errors for the same problem is beyond me, but that’s not important here. What is important is:
Wkhtmltopdf and Headless Chrome handle the Access Denied and Canceled Requests differently.
Wkhtmltopdf fails on the Access Denied on East, but will ignore Canceled requests on West and generate a PDF with our default font, Roboto.
Headless Chrome times out on the Canceled Request on West, but is more forgiving on the Access Denied on East and will generate a PDF with our default font, Roboto.
So there you have it. A rabbit hole that was kind of interesting and maybe this helps someone else out there as well. Thanks for reading.