The world's top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.

OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.

TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.

OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.

However, according to TollBit's findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to "bypass" robots.txt in order to retrieve or scrape all of the content from a given website or page.

A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions "into account each time we train a new model." A spokesperson for Anthropic did not respond to emails seeking comment.

Robots.txt is a single bit of code that's been used since the late 1990s as a way for websites to tell bot crawlers they don't want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.

A family in Naples, Florida, whose home was struck by debris that fell to Earth from outer space and punched a hole in the roof is pursuing $80,000 from Nasa in compensation for damages.

The law firm Cranfill Sumner said in a press release that it filed a claim on behalf of plaintiff Alejandro Otero and his family.

A metallic cylinder slab from a cargo pallet that had been released by the International Space Station in 2021 hit the Otero family home on 8 March 2024 while their son Daniel was home. No one was injured, though it created a hole in the roof and floor.

Otero told Wink News that the object almost hit his son, who was two rooms over.

The US space agency later confirmed the debris was from its flight support equipment. A section of the debris remained intact rather than disintegrating after it entered Earth’s atmosphere before falling to the surface.

Médecins Sans Frontières / Doctors Without Borders (MSF) teams in Gaza are facing critical shortages of essential medicines and equipment, having been unable to bring any medical supplies into Gaza since the end of April.

The closure of the Rafah crossing, following Israel’s offensive in the south of Gaza in early May, coupled with the endless red tape imposed by Israeli authorities, have dramatically congested the flow of humanitarian aid through the remaining Kerem Shalom entry point.

This has led to massive queues of trucks and perilous delays in the delivery of humanitarian assistance across Gaza.

And, even when aid can finally enter the Strip, insecurity often does not allow humanitarian organisations to get it to where it is desperately needed.

Despite medical needs skyrocketing across the Strip, without a significant refill of medical supplies in the coming days, MSF might have to stop or drastically reduce some of its medical activities in Gaza.

Guillemette Thomas, MSF medical coordinator in Palestine, says:

“Our medical supplies are critically low due to the limited flow of aid that is being allowed into Gaza by the Israeli authorities.

"If we don’t manage to get medical supplies into Gaza very soon, we may have to stop our medical activities. This is an unthinkable reality given the desperate medical needs of thousands of people in Gaza.

"We have patients with severe burns, open fractures, and we don’t even have enough painkillers to alleviate their suffering.

"In Nasser and Al Aqsa hospitals, our teams have had to reduce the frequency of dressing changes for patients with severe burns due to the lack of sterile compress gauzes, which could lead to more infected wounds.

TikTok says it offered the US government the power to shut the platform down in an attempt to address lawmakers' data protection and national security concerns.

It disclosed the "kill switch" offer, which it made in 2022, as it began its legal fight against legislation that will ban the app in America unless Chinese parent company ByteDance sells it.

The law has been introduced because of concerns TikTok might share US user data with the Chinese government - claims it and ByteDance have always denied.

TikTok and ByteDance are urging the courts to strike the legislation down.

"This law is a radical departure from this country’s tradition of championing an open Internet, and sets a dangerous precedent allowing the political branches to target a disfavored speech platform and force it to sell or be shut down," they argued in their legal submission.

They also claimed the US government refused to engage in any serious settlement talks after 2022, and pointed to the "kill switch" offer as evidence of the lengths they had been prepared to go.

Columbia University professors Awi Federgruen and Ran Kivetz have presented research findings asserting that sufficient food supplies are entering Gaza, disputing claims of famine perpetuated by the International Criminal Court (ICC) and the United Nations (UN, The Jerusalem Post reported on Tuesday.

Professors Federgruen, Chair of Columbia University Business School’s Decision, Risk and Operations Division, and Kivetz, the Philip H. Geier Professor at Columbia University Business School, have analyzed extensive data from sources such as COGAT (Coordinator of Government Activities in the Territories) and the UN.

They argue that the narrative blaming Israel for causing famine in Gaza is a "myth" and that sufficient humanitarian aid is being provided.

They pointed to a March 2024 report by the Integrated Food Security Phase Classification (IPC), an arm of the UN, which predicted a major famine in Gaza allegedly provoked by Israel. This report, they argue, has been misinterpreted and used to propagate false accusations.

Federgruen and Kivetz emphasize that their analysis is based on hard data, demonstrating that the food supply entering Gaza is more than sufficient to meet the needs of its 2.2 million residents. They calculate that 250 truckloads, each carrying 20 tons of food, provide 2.25 kilograms of food per person daily, aligning with the average North American diet.

Sixteen years ago, Riley Livermore enlisted in the Air Force. His path to the military was shaped by his evangelical Christian upbringing and growing up amid the war on terror. His ensuing career as a flight test engineer took him to Israel, where he spent two years doing missile guidance research. And shortly after October 7, he decided he couldn’t continue anymore.

Livermore is “utterly dismayed” by how President Joe Biden and the Department of Defense “has been complicit in the genocide in Gaza,” he told The Intercept. So much so that he is in the final steps of separating from the Air Force, a monthslong process he initiated in late October. Once he officially exits the military, he said, he will never again work in what he describes as the military–industrial complex.

“I don’t want to be working on something that can turn around and be used to slaughter innocent people,” he said. “I think the dissonance just kind of continued to get louder and louder, it’s like ‘I can’t really do this anymore.’”

Livermore joins a burgeoning wave of dissent within the Biden administration and the military over U.S. support for Israel’s war on Gaza — including nine prominent resignations in recent months; 25-year-old Airman Aaron Bushnell’s self-immolation in February; and a new service member-led campaign to help soldiers speak out against elected officials’ support for Israel’s war.

The University of Minnesota paused the hiring of a professor who wrote that Israel’s military operation against Hamas in Gaza after Oct. 7 was “a textbook case of genocide” to head the school’s Center for Holocaust and Genocide Studies (CHGS), Jewish Insider has learned.

The pause, which has not yet been publicly announced by the university, came on Monday evening after two members of the center’s advisory board resigned in protest on Friday.

“The assault on Gaza can also be understood in other terms: as a textbook case of genocide unfolding in front of our eyes,” Raz Segal, an Israeli associate professor of Holocaust and Genocide Studies at Stockton University in New Jersey, wrote in the Jewish Currents on Oct. 13. “I say this as a scholar of genocide, who has spent many years writing about Israeli mass violence against Palestinians,” he wrote.

A spokesperson for the University of Minnesota told JI that the director selection process was put on hold “to allow an opportunity to determine next steps.”

Clearview AI, a facial recognition start-up that scraped more than 30 billion photos from social media, can’t afford to pay the settlement bill from its class-action lawsuit so is offering Americans a stake in its company instead.

The New York-based company was sued in a federal court in Chicago in litigation that has proven so costly that Clearview AI says it will go bankrupt before making it to trial.

The unusual settlement offers members of the lawsuit a collective 23 percent stake in Clearview AI; an amount that is currently valued at $52 million, per The New York Times.

Members of the class could be literally anyone in the United States who has posted a photo of themselves online so anyone who does submit a claim could be entitled to a cut of the proceeds from a public flotation or an acquisition.

The UN agency for Palestinian refugees (UNRWA) reported on Saturday that over 50,000 children in the Gaza Strip are in urgent need of treatment for acute malnutrition, Anadolu Agency reports.

In a statement, the agency said that “with continued restrictions to humanitarian access, people in Gaza continue to face desperate levels of hunger.”

“Over 50,000 children require treatment for acute malnutrition,” it added.

Around 9:30 p.m. in late February, a white Mazda pulled up near a game cafe in the Jenin refugee camp on the northern edge of the West Bank, where a crowd of boys and young men often gathered to socialize.

As the car stopped, a few people walked by on the narrow street. Two motorbikes weaved past in different directions. “Everything was fine at the time,” according to an eyewitness sitting nearby in the camp’s main square.

Then the car erupted in a ball of flame. Two missiles fired from an Israeli drone had hit the Mazda in quick succession, as shown in a video the Israeli Air Force posted that night.

According to the IAF, the strike killed Yasser Hanoun, described as “a wanted terrorist.”

But Hanoun was not the only fatality: 16-year old Said Raed Said Jaradat, who was near the vehicle when it was hit, sustained shrapnel wounds all over his body, according to documentation collected by Defense for Children International-Palestine. He died from his injuries at 1 a.m. the next morning.

Jaradat is one of 24 children killed in Israel’s airstrikes on the West Bank since last summer, when the Israeli forces began deploying drones, planes, and helicopters to carry out attacks in the occupied territory for the first time in decades.

May 5, 2024, marked the first-ever leak of the most comprehensive collection of Google Search API ranking factors in the history of search engines – a truly historical moment we might not have seen if Erfan Azimi, founder & CEO of an SEO agency, hadn't spotted Google's documents that were mistakenly released on Github on March 27, 2024, and were just forgotten to be deleted. The irony is that they were published under the Apache 2.0 license, allowing anyone accessing the documents to use, edit, and distribute them. As such, sharing the documents with two of the most reputable SEO experts, Rand Fishkin and Mike King – the next step Erfan Azimi took after spotting the leak – was within the legal boundaries of the license. Both released the documents and accompanying analysis on May 27.

Navboost is a Google ranking algorithm that was revealed during the company's antitrust trial with the U.S. Department of Justice. It enhances search results for navigation queries by utilizing various signals like user clicks to identify the most relevant outcomes. Navboost retains past clicks for queries up to 13 months old and differentiates results based on localization and device type (mobile or desktop). This ranking signal is crucial for SEO professionals to understand and optimize for, as it can significantly impact a website's visibility in search results. Clicks are a primary ranking signal, indeed

Google has denied for years that clicks belong to a primary ranking factor. Its representatives, including Gary Illyes, have consistently emphasized that click-through rate (CTR) is a "very noisy signal" and that using clicks directly in rankings would be problematic due to the potential for manipulation. They have explained that while click data is used for evaluation and experimentation purposes to assess changes in the search algorithm, it is not a primary factor in determining search rankings.

The leaked documents prove otherwise. It does matter how many clicks a website can generate. The more on-page optimization and continuous content marketing you do, the more traffic you'll attract, resulting in more clicks, higher rankings, and higher conversion rates.

Google representatives have consistently misdirected and misled us about how their systems operate, aiming to influence SEO behavior. While their public statements may not be intentional lies, they are designed to deceive potential spammers—and many legitimate SEO professionals—by obscuring how search results can be impacted. Gary Ilyes, an analyst on the Google Search Team, has reiterated this point numerous times. He's not alone; John Mueller, Google's Senior Webmaster Trends Analyst, and Search Relations team lead, once stated they don't have a website authority score.

However, as the data leak suggests, Google does have an overall domain authority measure. As part of the Compressed Quality Signals stored on a per-document basis, Google computes a feature called "siteAuthority." According to Mike King, Founder and CEO of iPullRank, while this measure's specific computation and application in downstream scoring functions remain unclear, we now definitively know that Google's domain authority exists and is used in the Q* ranking system.

The recent Google Search API leak revealed the existence of white lists that are used to ensure the quality and reliability of information, particularly for sensitive topics like health and news, where misinformation could have drastic implications on public well-being.

An open letter from Apple employees and shareholders demands the tech giant stop matching employee donations to organizations with ties to the Israeli military assault on the Gaza Strip and ongoing illegal settlement development in the West Bank.

The letter, building on a previous demand by Apple employees for a ceasefire in the conflict, calls on the company to “promptly investigate and cease matching donations to all organizations that further illegal settlements in occupied territories and support the IDF.”

As with many large corporations, Apple employees can make donations to a number of nonprofit organizations and receive matching contributions from their employer through a platform called Benevity.

Among the charitable organizations eligible for dollar-matching from Apple are Friends of the IDF, an organization that collects donations on behalf of soldiers in the Israeli military, as well as a number of groups that contribute to the settlement enterprise in the West Bank, including HaYovel, One Israel Fund, the Jewish National Fund, and IsraelGives.