We have multiple apps using integrations. One of which is Location Importer. This app is used to convert Text columns to a Location column, a typical bulk action. monday limits users to bulk change 500 items at once but when you configure an automation to be triggered when “An item is created” you can literally trigger thousands of requests by importing an Excel or CSV.
For the app to work it has to perform some other API calls to the monday API such as retrieving the text value (query) and writing (mutation) the result back.
Especially with the mutations we run into complexity budget issues after we process the first 140-160 requests. Since the mutation request is at the end of our flow we have already performed other API calls such as increasing the operations (which is a cost for users) as well as using an external API to convert the text into an address (which is a cost for us).
We’ve introduced a complexity budget check before doing anything else to make sure we have enough “mutation” complexity budget available but due to concurrent requests we still see quite a lot of requests passing the complexity check but still running into complexity issues when performing the mutation.
We’ve also thought about using a queueing mechanism for the mutations itself but that would mean we can’t pass back any feedback to the user if anything goes wrong (as we already responded to the integration request with a 200 response).
Furthermore the Short Lived Token is only valid for 5 minutes so that means we would only be able to process between 700 - 800 items at once which is not enough in many cases, if we would go over this maximum, we will run into expired token issues without any ability to report back to the user or ask for a retry request from the monday backend.
Therefore we are suggesting the ability to configure a maximum amount of request your integration actions can process per minute. If monday would take those maximums into account we will:
Avoid unnecessary retries from monday towards our app (when we first responded with an error because of the complexity issues)
Avoid unnecessary users & developer costs
Avoid the maximum limit of 800 items when using a queue
If you agree or run into the same issues, please vote for this feature request!
Some numbers
Without the complexity check built in we were actually processing 900+ requests for just 500 action triggers (80% redundant calls)
With the complexity check built in we are still processing 650+ requests for just 500 actions (30% redundant calls)
Thanks for the reference Cody, they do indeed relate. And I completely agree that monday’s queue for integrations is the best queue to use instead of implementing our own queue
On a side note (also for @anon29275264). I just found out this week (the hard way) that monday is maintaining a “secret” list of apps that did “too many” retries in the past (don’t ask me how many “too many” is ). When your app hit that list monday will simply ignore retry requests (429 responses) from the app. Because the list is “secret” nobody knows about this and only 3 weeks of support email back and forth with the customer and monday support, support will kindly tell about this list and remove the app from it… what a laugh
@dipro just calling out that the situation @basdebruin is describing is asinine.
We’re going to be punished for retrying requests because of monday’s low API rate limits (mutating 150 items per account per minute or there about, maybe 162 depending on the exact number of columns).
We’re supposedly supposed to queue requests - but to do that we have to return “success” to the request (as we’ve discussed many times over) which is going to lead to multiple problems - the least of which is erroneous activity log entries. The worst of which is with a custom automation sentence, queuing and returning success will cause the next action to execute - even though the prior action is not complete and may need the values changed by the queued action.
Monday is literally forcing us to create race conditions for customers. This seriously undermines monday.com’s credibility as an enterprise platform.
Hi @basdebruin & @anon29275264 - Our focus is transparency in all aspects, so I would like to better address why this specific feature flag was added to your account. The intention was never to be vague & for that, we do apologize.
This specific feature flag was granted to the app in response to the app not adhering to the prescribed error handling guidelines (Error handling). Unfortunately, the guidance was not followed when the error was not retryable and caused an influx of errors crowding our automations queue. To mitigate this, we instated a feature flag blocking further retries for this app.
This in mind, we agree that the communication of this decision fell short. It’s important to us that no one is left in the dark about such changes, and we assure you, we’re revisiting our communication process to better execute such announcements in the future.
Just to clarify, I am totally separate from Bas (as far as apps).
My apps return a 429 when we are up against API complexity limits on the monday side of things. We would queue on our side, except to do so we have to return a success to monday. This means we cannot report actual failures for what is now queued - the customer always sees success. This confuses them (trust me, we used to do this!)
It also means next steps in automations will start running before we’ve completed our queued actions and that will confuse end users with unpredictable behavior/results for their workflows.
How are we supposed to handle it in a way that has consistent and predictable results for our customers, provides them accurate activity feedback, and doesn’t risk getting our app flagged as misbehaving in some way?
How can we make sure which of our current apps are on that list. It has never been communicated and working through support took over 2 month to find out.
When an integration app is triggered by a user that selects e.g. 2.000 items on a board and make a change to a column the app will obviously receive a complexity error when trying to change a column value as a result of the trigger. The app sends a 429 and rely on monday retry mechanism. The retry will happen while we are still processing the queue so another 429 will be send. At the end monday will give up (understandably) but we don’t know if that is the trigger to put the app on this list.
Also, you are mentioning “specific feature flag was added to your account”, is that 'specific" feature account wide or app wide?
Agreed that this needs to be clearly documented and we plan to contact any apps where this flag was added. The flag is added per app, not per account. As the fix was targeting the behavior of specific apps, not an entire account.
Again, it is on us for not communicating this and efforts to increase transparency have always been top priority.
If you do have a specific technical question on the app or error handling, please open a technical request with our team and they will be able to help!
Thank you for the follow up here Cody! I know you all have separate apps, but the challenge you are both facing is similar so I wanted to address both of your concerns around preventing retries together. Although none of your apps were granted this feature blocking retries(we will be communicating with all affected apps as soon as possible), the issue with reporting failures, providing consistent behavior for customers, and maintaining seamless service are the same.
I can see you’ve opened several feature requests so I would recommend promoting those within the community with other developers. I would also recommend reaching out to our technical team via a support request if you have a specific situation or concern we can take a closer look at with you! It can be very helpful to have context of a specific situation within the context of your app so our team can make the best recommendations from there.
Mostly I want to avoid ending up on that naughty list! So I await communication about the criteria of when that flag gets added, etc. so I can try to avoid it.
). When your app hit that list monday will simply ignore retry requests (429 responses) from the app. Because the list is “secret” nobody knows about this and only 3 weeks of support email back and forth with the customer and monday support, support will kindly tell about this list and remove the app from it… what a laugh