|
 |
| | | Author: | Rakali  | | Posted: | Oct 9, 2021 04:59 | | Subject: | Programmers: Page reqeust limit help. | | Viewed: | 92 times | | Topic: | Related Software | |
|
| First let me say that I am not a programmer, not even a little bit, and what
code I have so far is cobbled together from stack overflow and is extremely janky.
I'm just trying to make a little program for myself where I can compare LEGO
B&P prices of a wanted list to a BL stores prices.
I can pull the prices from B&P, and they don't seem to have any sort of lock
on how many page requests I make or how fast, but when I try to do the same from
a BL store, I'm running into 403 errors half of the time, and eventually
getting completely locked out to the point where I cannot view the inventory
of any shop even within my browser.
I'm not exactly flooding the server with requests here, a wanted list of
~100 items with half a second wait between each request gets me locked out straight
away so it has to be pretty strict.
Is it documented anywhere how many requests are allowed to be made, or is there
a better way to be doing what I am trying to do than loading each page individually
(and using some jank regex to grab the price).
Cheers for any advice!
|
|
|
| | | | | |
| | | | | Author: | BlocShop | | Posted: | Oct 9, 2021 05:42 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 39 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
| | First let me say that I am not a programmer, not even a little bit, and what
code I have so far is cobbled together from stack overflow and is extremely janky.
I'm just trying to make a little program for myself where I can compare LEGO
B&P prices of a wanted list to a BL stores prices.
I can pull the prices from B&P, and they don't seem to have any sort of lock
on how many page requests I make or how fast, but when I try to do the same from
a BL store, I'm running into 403 errors half of the time, and eventually
getting completely locked out to the point where I cannot view the inventory
of any shop even within my browser.
I'm not exactly flooding the server with requests here, a wanted list of
~100 items with half a second wait between each request gets me locked out straight
away so it has to be pretty strict.
Is it documented anywhere how many requests are allowed to be made, or is there
a better way to be doing what I am trying to do than loading each page individually
(and using some jank regex to grab the price).
Cheers for any advice!
|
I think its best if you would use the bricklink api and not some regex solution
for this: https://www.bricklink.com/v2/api/welcome.page
Beware that there is a daily limit of 5000 request
|
|
|
| | | | | | | | | |
| | | | | | | Author: | Teup | | Posted: | Oct 9, 2021 06:10 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 31 times | | Topic: | Related Software | |
|
| In Related Software, MXbricks writes:
| | In Related Software, Rakali writes:
| | First let me say that I am not a programmer, not even a little bit, and what
code I have so far is cobbled together from stack overflow and is extremely janky.
I'm just trying to make a little program for myself where I can compare LEGO
B&P prices of a wanted list to a BL stores prices.
I can pull the prices from B&P, and they don't seem to have any sort of lock
on how many page requests I make or how fast, but when I try to do the same from
a BL store, I'm running into 403 errors half of the time, and eventually
getting completely locked out to the point where I cannot view the inventory
of any shop even within my browser.
I'm not exactly flooding the server with requests here, a wanted list of
~100 items with half a second wait between each request gets me locked out straight
away so it has to be pretty strict.
Is it documented anywhere how many requests are allowed to be made, or is there
a better way to be doing what I am trying to do than loading each page individually
(and using some jank regex to grab the price).
Cheers for any advice!
|
I think its best if you would use the bricklink api and not some regex solution
for this: https://www.bricklink.com/v2/api/welcome.page
Beware that there is a daily limit of 5000 request
|
However, the API is slow and if it's part-out values you're after, those
are not available in the API. Also, the API is very complicated compared to BrickOwl's
- it's probably not feasable for the OP unless they get some help with it
(like I did).
I've ran into the same problem, Bricklink simply doesn't want you to
do it, at all. Which is clearly reasoned from a non-BL-seller perspective. I've
ended up simply adapting my workflow to that reality. Maybe you can design things
in a way you can process smaller batches spread out over several days, and cache/save
everything you've got to minimize how much you need.
|
|
|
| | | | | | | | | | | | | |
| | | | | | | | | Author: | Rakali | | Posted: | Oct 9, 2021 16:55 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 30 times | | Topic: | Related Software | |
|
| | | However, the API is slow and if it's part-out values you're after, those
are not available in the API. Also, the API is very complicated compared to BrickOwl's
- it's probably not feasable for the OP unless they get some help with it
(like I did).
I've ran into the same problem, Bricklink simply doesn't want you to
do it, at all. Which is clearly reasoned from a non-BL-seller perspective. I've
ended up simply adapting my workflow to that reality. Maybe you can design things
in a way you can process smaller batches spread out over several days, and cache/save
everything you've got to minimize how much you need.
|
I can totally understand why they don't want people mass scraping data, that's
why I'm just keen to know the limits so I can abide by them.
|
|
|
| | | | | | | | | | | | | | | | | |
| | | | | | | | | | | Author: | Teup | | Posted: | Oct 10, 2021 04:42 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 44 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
| | | | However, the API is slow and if it's part-out values you're after, those
are not available in the API. Also, the API is very complicated compared to BrickOwl's
- it's probably not feasable for the OP unless they get some help with it
(like I did).
I've ran into the same problem, Bricklink simply doesn't want you to
do it, at all. Which is clearly reasoned from a non-BL-seller perspective. I've
ended up simply adapting my workflow to that reality. Maybe you can design things
in a way you can process smaller batches spread out over several days, and cache/save
everything you've got to minimize how much you need.
|
I can totally understand why they don't want people mass scraping data, that's
why I'm just keen to know the limits so I can abide by them.
|
They just don't want us to do it, at all - the limit is 0  It is somewhere
in the terms.
I think it's like, whatever informal limit is there, is just there to detect
that you are in the process of scraping, and then they kick you out. In my experience
it varies, I recall a time I was not able to retrieve even 10 part out values,
without having done anything else that day.
|
|
|
| | | | | |
| | | | | Author: | yorbrick | | Posted: | Oct 9, 2021 06:26 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 38 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
| | First let me say that I am not a programmer, not even a little bit, and what
code I have so far is cobbled together from stack overflow and is extremely janky.
I'm just trying to make a little program for myself where I can compare LEGO
B&P prices of a wanted list to a BL stores prices.
I can pull the prices from B&P, and they don't seem to have any sort of lock
on how many page requests I make or how fast, but when I try to do the same from
a BL store, I'm running into 403 errors half of the time, and eventually
getting completely locked out to the point where I cannot view the inventory
of any shop even within my browser.
I'm not exactly flooding the server with requests here, a wanted list of
~100 items with half a second wait between each request gets me locked out straight
away so it has to be pretty strict.
Is it documented anywhere how many requests are allowed to be made, or is there
a better way to be doing what I am trying to do than loading each page individually
(and using some jank regex to grab the price).
Cheers for any advice!
|
If you have the wants list and are in the store, why not just use the wants list
to search that store. For 100 parts it is not that onerous. Or do you plan to
do it in multiple stores?
|
|
|
| | | | | | | | | |
| | | | | | | Author: | Rakali | | Posted: | Oct 9, 2021 16:52 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 34 times | | Topic: | Related Software | |
|
| | |
If you have the wants list and are in the store, why not just use the wants list
to search that store. For 100 parts it is not that onerous. Or do you plan to
do it in multiple stores?
|
The plan is to do it for multiple stores so I can choose the stores I want to
buy from as well as compare to B&P, that way I can make my own decisions instead
of having the auto-finder recommend me stores that are manipulating prices using
fees and other such shenanigans.
|
|
| | | | | | | | | | | | | |
| | | | | | | | | Author: | SylvainLS | | Posted: | Oct 9, 2021 16:54 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 21 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
| | | |
If you have the wants list and are in the store, why not just use the wants list
to search that store. For 100 parts it is not that onerous. Or do you plan to
do it in multiple stores?
|
The plan is to do it for multiple stores so I can choose the stores I want to
buy from as well as compare to B&P, that way I can make my own decisions instead
of having the auto-finder recommend me stores that are manipulating prices using
fees and other such shenanigans.
|
Or you can least-favorite stores and exclude them from Auto-select….
|
|
| | | | | | | | | | | | | |
| | | | | | | | | Author: | yorbrick | | Posted: | Oct 9, 2021 17:11 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 28 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
| | | |
If you have the wants list and are in the store, why not just use the wants list
to search that store. For 100 parts it is not that onerous. Or do you plan to
do it in multiple stores?
|
The plan is to do it for multiple stores so I can choose the stores I want to
buy from as well as compare to B&P, that way I can make my own decisions instead
of having the auto-finder recommend me stores that are manipulating prices using
fees and other such shenanigans.
|
You will still need to look up that type of thing manually. If they arexadding
on extra fees in their terms you won't be able to easily automate it.
It is probably easier to use wants lists but least favourite any stores manipulating
fees.
|
|
|
| | | | | | | | | | | | | | | | | |
| | | | | | | | | | | Author: | Rakali | | Posted: | Oct 10, 2021 00:11 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 28 times | | Topic: | Related Software | |
|
| | | It is probably easier to use wants lists but least favourite any stores manipulating
fees.
|
Sadly at this point that is not an easy feat. What I want to do is start with
stores I would like to buy from (reasonable fees, decent pricing) first, and
then see what parts they have, and go from there.
For reference my test project is 90 something lots, 84 of those are available
on B&P, so Im manually finding a store to buy the missing lots from.
Then because I am buying from that store anyway, I would like to compare the
rest of the lots in the list to see if they have any cheaper than B&P so I can
pick them up as well while I am there.
|
|
| | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | Author: | yorbrick | | Posted: | Oct 10, 2021 01:30 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 25 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
| | | | It is probably easier to use wants lists but least favourite any stores manipulating
fees.
|
Sadly at this point that is not an easy feat. What I want to do is start with
stores I would like to buy from (reasonable fees, decent pricing) first, and
then see what parts they have, and go from there.
For reference my test project is 90 something lots, 84 of those are available
on B&P, so Im manually finding a store to buy the missing lots from.
Then because I am buying from that store anyway, I would like to compare the
rest of the lots in the list to see if they have any cheaper than B&P so I can
pick them up as well while I am there.
|
It might be worth just checking average prices on BL vs B+P. If the average is
cheaper here, chances are you will find a seller with it cheaper here.
I also learnt years ago doing this sort of thing manually, don't get hung
up on pennies unless you are buying 100s of an item. Spending a few hours to
save £1 on a £100 MOC is not time well spent.
|
|
|
| | | | | |
| | | | | Author: | cosmicray | | Posted: | Oct 9, 2021 11:32 | | Subject: | Re: Programmers: Page reqeust limit help. | | Viewed: | 42 times | | Topic: | Related Software | |
|
| In Related Software, Rakali writes:
You do realize that the Price Guide tab gives you a wealth of information, and
all for a single request. If you did those 45-60 seconds apart, you could collect
considerable data with a minimal number of requests.
Nita Rae
|
|
|
|
|
United Kingdom, England
Netherlands, Zuid-Holland
France, Nouvelle-Aquitaine
USA, Florida