Help in Scraping DATA using OPENBULLET
-
Hi guyz
i'm scraping data using an API that not allows multiple requests at same time using same ip (Scraping is legal but there is limits)
After the API response, bot should wait 1 sec before next request using same ip
i found no bypass for this limit
i set retry key, proxy will change after 1 retry, but i don't want make pressure on target with that unreasonable requestsis there any way that i could set proxy status as Bad/Banned after Success/Custom/Fail status?
i tried SET PROXY commands and OB/Config settings, i found no way
-
BAN = retry data and ban the proxy
RETRY = retry data with the same proxy
Also you can use the Function > Delay block
-
If you want to ban a proxy after SUCCESS you can do that in OB2, not in OB1
-
@Ruri Can we create new custom STATUS in OB2, like we make custom wordlists using [Environment.ini]?
i need a DONE instead of SUCCESS, or SUCCESS without saving Creds/Data in DB/TXTand also i don't wanna see Creds/Data and the captured data in the Runner UI / Or only see limited number of hits in runner
both needed for data scraping and offline tools that i made using OB
-
@Ruri said in Help in Scraping DATA using OPENBULLET:
If you want to ban a proxy after SUCCESS you can do that in OB2, not in OB1
No,
Actually i wanna set proxy as BAD after SUCCESS or even FAIL, i don't want to ban that proxy
-
Bad is treated as Banned, it's the same thing, I don't even know why I made a distinction.
-
Anyways, about OB2 config settings
and yes you can expand them if you edit the Environment.ini file
-
@Ruri said in Help in Scraping DATA using OPENBULLET:
Bad is treated as Banned, it's the same thing, I don't even know why I made a distinction.
So u don't remember
Bad proxy will go to active proxies after all proxies goes to BAD
But BANNED proxies will only reactivated after all proxies bannedwith Retry (in KEYCHECK) we can set proxy as BAD and with BAN (in KEYCHECK) we can set proxy as BAN
BAD is good to make proxies rotating, and BAN is good to deactivate dead proxies for long time
two loops here works good
and that's actually very useful
-
@Ruri said in Help in Scraping DATA using OPENBULLET:
Anyways, about OB2 config settings
and yes you can expand them if you edit the Environment.ini file
"Ban the proxy when the bot ends with one of these statuses" is good, but if you also add another option as "consider the proxy as Bad when the bot ends with one of these statuses" is better
