I’m running into an interesting issue while performing heavy queries.
If the call takes longer than 60 seconds, I get one of two responses from the server. Both are errors, and both are in HTML body. One is a general 504 timeout warning. The second one is returned the page text below. After watching the timing, this happens at exactly the 60 second mark, so the timeout seems to be 60 seconds from the server. I’m not seeing this mentioned anywhere on the documentation.
This exposes a few critical issues and opportunities.
If the anticipated API response body is JSON, ALL API response bodies should be JSON. Every. Single. One. I think it is inappropriate to return a page body when there’s an error, instead of simply {“error”:“Server timeout. Maximum query duration is 60 seconds.”} or something similar displaying the reason for the error. We can always manage exceptions, however response body format should never vary like this.
I’m disappointed further that pagination doesn’t interrupt query timeouts. Return an error at data transmission if anticipated query call will result in greater than 60 seconds, or just complete the query and send the data.
I have noticed similar issues receiving a 504 Timeout Error when trying to retrieve large queries. You bring up very good points about issues and possible opportunities.
I would like to share something that I discovered about query requests that I have not been able to find in the documentation either:
When passing the limit parameter to a query response, subsequent queries to the same board (e.g. within a looping structure) will select the next contiguous block of items from the board, instead of starting from the beginning of the board. I haven’t been able to confirm this, but it seems that Monday.com maintains an internal cursor when a query returns a limited number of items.
When working with large boards (either a large number of items, or a large number of columns), I have found that the ability to “break up” the data that comes from the query solves the problem of the request timeout, and then I can concatenate all results locally.
@arnold_wikey yes! I’ve found the same. The two seem to work together so long as they are at the root level of your request. I’m currently pulling full pulse sets across all boards the way you are describing above. I had some weird responses having page requests anywhere except at root during testing that was a bit annoying. For example, I couldn’t successfully pull board --> item with page on item, and of course page on board just breaks up the boards. I think the pagination documentation could use some more description regarding when it does and does not work, as well as documentation regarding time outs and Monday’s recommendation for avoidance.
there are API rate limits on the complexity levels which are 10,000,000 / minute per account on monday.com.
It’s higly recommended to lower the complexity level of your querys.
@TMNXT-Dev I appreciate your response, however I think we are discussing different topics.
I record the cost of every query I run.
From what I see query costs do not align with machine workload
This is the most expensive query I run. It costs 2,013,010 and runs in less than a second. It costs so much because I pulled a lazy move and didn’t feel like cycling through board page, and Monday’s cost calculator multiplies by 1,000. Regardless, here it is.
Any limit over 2000 on these item requests pushes response time too closely to the 60 second mark for comfort. When I adjust my limit to 4000 or 5000 I bump into response times that are 60+ seconds and run into timeout issues, with a minimal query cost per run time. Here’s an example, using a page number higher than what currently exists in my dataset. Minimal cost, but if the data existed the call is likely to timeout.
When a timeout is achieved, the response is an html page body, instead of a json body with an error code. This add unnecessary complication to error handling.
Thank you so much for taking the time to bring this to our attention. The errors being returned in HTML as opposed to JSON is something that we’re aware of and are working on resolving. I’ve passed this feedback along to the team to let them know it’s a pain point that you’re currently running into.
You’d also noted:
I’m disappointed further that pagination doesn’t interrupt query timeouts. Return an error at data transmission if anticipated query call will result in greater than 60 seconds, or just complete the query and send the data.
I just wanted to let you know that I’ve passed that feedback along to the team as well
If you have any other suggestions, please let me know!
@morhriveion@arnold_wikey how have you guys been calculating the total pages? I’m running into a similar issue of timeout and was hoping I could loop through the items locally. Any tips or tricks? Thanks guys!
@morneluckybeard my use may be different from yours depending on the amount of columns you are calling at a time.
I used a variable to control how many rows/pulses I called at a time and dropped that down until the highest call wast no more than 20-30 seconds. I left that variable at 2000.
I don’t remember why I chose to not check for a response equal to the variable. Instead I opted to call the next page so long as the previous call returned pulses. I didn’t leave any notes on it but I’m sure either would work.
The time out issue was my biggest frustration while working with this. Definitely make sure you are not at risk of bumping into this timeout. During my first test phase I was around the 4000-5000 mark, and had zero issues. Unfortunately I do a lot of work during non-peak hours, and once I moved into any where near peak hours I ran into timeouts that broke every request sequence.
Thanks for taking the time to respond, I was hoping I could kick off all of my calls asynchronously instead of calling them sequentially, but I think I might have to do that, reduce the limit to like 200 and keep calling until I get an empty array back.
.
