Big Data and a Saturation Point

Click For Summary

Discussion Overview

The discussion revolves around the concept of "big data" and the potential for a saturation point in the demand for data processing and storage. Participants explore the implications of increasing data complexity on computing architectures and algorithms, touching on themes such as data science, cloud computing, and the limitations of current technologies.

Discussion Character

  • Exploratory
  • Technical explanation
  • Debate/contested

Main Points Raised

  • One participant suggests that big data represents a significant cultural shift in the information age, raising questions about the potential saturation point in data demand.
  • Another participant expresses skepticism about the existence of a saturation point, arguing that modern processors can handle vast amounts of data efficiently, potentially only requiring a statistically meaningful subset for analysis.
  • A participant inquires about existing or anticipated pressures on current computing capabilities related to data processing.
  • It is noted that some algorithms may only achieve rapid solutions with advancements in quantum computing, indicating limitations in current non-quantum processing capabilities.

Areas of Agreement / Disagreement

Participants do not reach a consensus on the existence of a saturation point in data demand, with differing opinions on the implications of current computing capabilities and future technological advancements.

Contextual Notes

The discussion includes assumptions about the definitions of "saturation point" and the capabilities of current computing technologies, which remain unresolved.

Gear300
Messages
1,209
Reaction score
9
I ask this with more of a software background than an engineering background, but here I go anyway.

Big data is arguably the cultural motif or monograph of the information age. Trends involve immersing ourselves in media of various sorts and processing them at exceptional rates. Of course, the information could be poisoned with false info, propaganda, virulent memes, and other redundancies, but at the very least, none of this is ahistorical. Other trends include the fourth paradigm in science, or data science; business compaction and migration to the cloud; streaming data, or data that exists in the network and not just at endpoints; and so on.

So the question here has to do with the growing demand and complexity of data. As it grows, it grates against our architectures and algorithms. The epitome of classical computing lies with parallel architectures, both horizontal and vertical, e.g. microservices in industry solutions. Big cloud vendors run on networks with a bisection bandwidth on the scale of petabytes, and they do what they can to optimize data storage and processing. But is there a credible futurist or tech-economist who expects there to be a saturation point in the growing demand for data? Sort of like one that prefigures the intermittency of classical computing in the early information era :biggrin:?
 
Last edited:
Engineering news on Phys.org
Depends on how you define "saturation point".

Personally, I find it hard to see why there would be. Even if you have a totally serial process with scadzillions of data points, given the speed of modern processors it would have to be an awfully slow/complex algorithm for it to take more time that it is worth to run. Worst case, it seems to me, would be that you have to use a statistically meaningful subset of the data.
 
  • Like
Likes   Reactions: russ_watters
I kind of figured that was the case, but thought it was worth asking. Do you know of any incumbent pressures on our current computing, current or expected, that could be listed?
 
Not in terms of data. As for algorithms, there are many for which it is believed that rapid solution will be possible only after we have much more robust quantum computing systems than are currently available. Solution by current non-quantum processors is completely out of the question.
 
Alright. I guess I could read up on the rest. Thanks for the reply.
 

Similar threads

Current and Future AI Threats to Humanity and the Human Response
  • · Replies 10 ·
Replies
10
Views
5K
Requesting suggestions for languages, libraries, and architectures for parallel (and sometimes non parallel) numerical and scientific computations
  • · Replies 8 ·
Replies
8
Views
5K
Mathematica This Week's Finds in Mathematical Physics (Week 247)
  • · Replies 1 ·
Replies
1
Views
4K