Welcome to the Q&A Forum

Unfortunately I'm not able to provide you a specific timeframe when links from Just-Discovered Links would appear in the Mozscape index. From the blog post linked above, Mozscape has it's own crawl scheduling timeframe.

Because of the time it takes to identify links, schedule, crawl and index them, it will typically be several weeks before links will appears in the Mozscape index. The high domain authority of linking domains will help ensure the links are included in the index, but do not help in speeding the process of getting links indexed.

I hope this helps!

Hi Thomas -

We are actively working on a campaigns API. This API will be for Moz Analytics, which is still in closed beta. I don't have a firm timeframe, but because of this dependency I wouldn't expect we would be able to launch this API before Q4.

Thanks for your continued interest in this data source, and for explaining your need. We're working on it!

Cheers,

Tela

Hi dealblogger!

That's a great question. At the moment, Just-Discovered Links is powered by an index that is completely separate from the Mozscape index. Links that are found and included in Just-Discovered Links are not automatically fed into our Mozscape index. I'll add your request to combine links from Just-Discovered with our primary index.

Links that we discover are crawled for inclusion in our main index based on the authority of those URLs. Here is a blog post that goes into detail about how Roger crawls the web and which URLs are included in our index. Thanks for writing-in!

Cheers,
Tela

Hi Adrian -

My apologies for the delayed response. We don't have any moral or philisophical objection to scraping. You are just trying to be more efficient about how you access your own data, after all.

The concern on our end lies entirely with how scraping could negatively impact other customers by creating a high load on our systems. Our own crawlers have built-in politeness capabilities that are conservative and we prohibit scraping because we can't ensure that others will do the same to us. If we gave blanket permission for scraping, this could lead to site performance or availability issues due to unpredictable loads.

The API is built to handle these larger and more condensed requests for data, so would be a natural choice. I'm actively working on a solution for your needs. This capability won't be ready before Q4 of this year. In fact, I would love to get your feedback on these features in a private thread.

I realize that the load you will generate will be low, but unfortunately I cannot grant permission for you to scrape this data.

Hi Adam -

I love this thread - thank you so much for starting this. I'm deeply interested in the ways that you and other customers would like to see this fresh data combined, how you would like to view it, and the time ranges that this information is useful for you. We launched these features separately to get customer feedback before creating a more integrated solution.

I'd enjoy hearing from other folks in the discussion group on this. Is a combined view of URLs/links and mentions valuable? Would you like to see it divided? Are hourly updates often enough, or even too often?

To set expectations, the scale of data that we are dealing with is challenging, so please understand I can't make any promises that features discussed here will get into the final product. We are still in the planning phase of different scenarios, and will continue to listen to customer feedback here and elsewhere to decide on the final product plan.

Thanks again, Adam and to all those who join this conversation.

Hi adriandg,

We have begun work on this as part of some more extensive work around our APIs. However, we likely won't be able to offer APIs for campaigns in the first half of this year.

I completely understand your wish to scrape the data yourself, and I really appreciate you asking permission first. Unfortunately we can't grant permission to scrape campaign data for a few reasons, including the load it generates. I would ask you to please wait for our API which will allow you to do this.

The good news is that we are actively working on this, and understand our customers' needs for this.

Tela

I agree with Dr. Pete that this seems like the same sort of challenge that affiliate links have, even if this isn't a standard affiliate program link. It sounds to me like there might be a UI/UX solution that could address this issue.

Could you add an intersticial URL / step after the user selects the phone they want that contains the tariff codes? Could you use a selector on the page to let the user select and generate the link with the tariff code dynamically, thus having one on-page URL conserving link juice? Just a couple of ideas, but it seems like the real issue is conserving link juice, and it's not necessary to keep the on-page links under the 100 link limit.

Hi Folks -

To give a little more clarity around the issue Google hasn't given us specific feedback on features that led to our credentials being removed. Without specific guidance, we would rather focus our efforts on getting replacement data from other sources.

Yvette Valencia - thanks for your suggestion. We are looking at solutions that would provide a work-around for power users but are still evaluating options.

Hi Folks -

To give a little more clarity around the issue Google hasn't given us specific feedback on features that led to our credentials being removed. Without specific guidance, we would rather focus our efforts on getting replacement data from other sources.

Yvette Valencia - thanks for your suggestion. We are looking at solutions that would provide a work-around for power users but are still evaluating options.

Hi dealblogger!

That's a great question. At the moment, Just-Discovered Links is powered by an index that is completely separate from the Mozscape index. Links that are found and included in Just-Discovered Links are not automatically fed into our Mozscape index. I'll add your request to combine links from Just-Discovered with our primary index.

Links that we discover are crawled for inclusion in our main index based on the authority of those URLs. Here is a blog post that goes into detail about how Roger crawls the web and which URLs are included in our index. Thanks for writing-in!

Cheers,
Tela

I agree with Dr. Pete that this seems like the same sort of challenge that affiliate links have, even if this isn't a standard affiliate program link. It sounds to me like there might be a UI/UX solution that could address this issue.

Could you add an intersticial URL / step after the user selects the phone they want that contains the tariff codes? Could you use a selector on the page to let the user select and generate the link with the tariff code dynamically, thus having one on-page URL conserving link juice? Just a couple of ideas, but it seems like the real issue is conserving link juice, and it's not necessary to keep the on-page links under the 100 link limit.

Unfortunately I'm not able to provide you a specific timeframe when links from Just-Discovered Links would appear in the Mozscape index. From the blog post linked above, Mozscape has it's own crawl scheduling timeframe.

Because of the time it takes to identify links, schedule, crawl and index them, it will typically be several weeks before links will appears in the Mozscape index. The high domain authority of linking domains will help ensure the links are included in the index, but do not help in speeding the process of getting links indexed.

I hope this helps!