InsightIQ Blog
The Real Time Database Complex
May 26 2009
Delivering data in "real time" is becoming an increasingly important design requirement for database architects. However, building a true real time database (where data is available the instant it is created or updated) is a complex design challenge when the database is not the system of record. Even if the database is indeed the system of record, the complexity of user queries often requires additional aggregation in order to be optimized for reporting, campaigning, analytics, etc.
Data within application databases typically goes through an ETL (extract, transformation and load) process. The ETL process includes several steps such as data cleansing and hygiene, matching, transaction processing, aggregation and auditing. This process takes time, whether it is two hours or two minutes, and therefore challenges the definition of real time. So what is a database architect to do?
The design tip below from Kimball University (part of the Kimball Group - a leading authority among data warehousing practices and techniques) on the "Real Time Triage" explains their recommended approach when collecting real time data requirements. The article explains a three tier process where data requirements are broken into the following categories: Instantaneous, Frequently, and Daily. Each category is explained in depth along with the benefits and tradeoffs.
http://www.kimballgroup.com/html/07dt/KU89TheRealTimeTriage.pdf
Leave a Comment
Ask Matt Austin A Question
Do you have a question for Matt Austin? Fill out the form below with your question.
View All Related Blog Posts
- Customer Data Management with "Big Data" - Part 1
- Building the Data Foundation, A Brief Interlude – Master Data Management
- Relax! We're watching for you.
- The Necessity of Data Profiling
Recent Comments
Blog Archive
- 2012 (6)
-
2011 (70)
- December (6)
- November (5)
- October (3)
- September (5)
- August (7)
- Are Communication Service Providers Really Ready for the Future?
- 3 Simple Ways to Build a Stronger Marketing Program
- 3 Ways to Stave Off Digital Copycats
- Direct Mail Darwinism: Surviving in the Digital Age
- Corporate Marketers Beware: Budgeting Changes Are Coming
- Engagement [en-geyj-muhnt]
- Quaero's Michelle Boockoff-Bajdek Featured in Recent DM News Article
- July (5)
- June (6)
- Bridging the Actionable Analytics Gap > Part2: Back to Basics
- 2 Critical B2B Marketing Insights from the MarketingProfs B2B Forum
- How the internet radio audience is changing, well, everything...
- Customer Insight Visualization at Warp Speed
- 4 Key Steps to Smarter B2B Marketing: Think Like a Database Marketer
- Customer Engagement Video: Data & Data Services
- May (7)
- Reporting from Forrester's Marketing Forum
- Role of analysis in customer engagement
- Turning audience data into revenue: Part II
- When Technology Fails: The Disconnected Customer Experience
- The Customer Engagement Agency: CEA Video Series
- Turning audience data into revenue: Part I
- B2B Sales and Marketing Alignment: Steps to Getting There
- April (6)
- Bridging the Actionable Analytics Gap
- Treat Your Customer Like You Would Treat Your Best Friend
- The clock is ticking on net tracking
- Quaero Founder Naras Eechambadi Featured in BtoB Magazine Virtual Roundtable
- Put Your Data Mining Skills to Work and Make $3 Million
- Salesforce.com Acquisition of Radian6
- March (7)
- Addressable TV advertising starting to go mainstream
- Quaero InsightIQ Now Featured on B2B Marketing Zone
- Increasing Marketing Complexity Demands Smart Outsourcing
- A standard for online engagement metrics - finally!
- ESPN: Maximizing Audience Data to Drive Revenue Growth
- The Massive Shift to Online Advertising Spend: Who is best poised to tap into that revenue?
- Building Your Data Foundation, Part II
- February (7)
- Building the Data Foundation, A Brief Interlude – Master Data Management
- Marketers Revisiting the Definition of a Best Customer
- Quaero Featured in Recent Internet.com Article on CRM Systems
- Building Your Data Foundation, Part I
- Customer Loyalty During Tough Times...
- ESPN's John Kosner: "You are what your record says you are"
- Four Steps to Successful Lead Scoring
- January (6)
-
2010 (95)
- December (5)
- November (7)
- How Chick-Fil-A Became My Favorite Fast Food Restaurant
- Analytics: Still Looking For A Home
- Online Analytics in Action: Footprints in the Dark
- Controversial Book on Amazon Sparks Intense Consumer Debate
- iPad Applications: So Many Opportunities
- INTERNET FAIL - The Today Show's "Steals & Deals"
- Click here, dear...
- October (5)
- September (7)
- Customer Analytics > Optimizing Collection Efforts, Part2
- The most magical place on earth - Walt Disney World
- Agile Marketing – Constant Evolution
- Metrics for e-Commerce
- Customer Surveys: Feedback is a Two Way Street
- Customer Engagement: Making It Happen
- Agile thinking will help marketers be more flexible
- August (8)
- Business Impact of Data Validation - Volume or Economic Value
- Make Room for Giants: The State of the Enterprise Marketing Management Space
- Control Yourself! Experimental Design for Marketing
- IBM’s acquisition of Unica will have long-term benefits for clients
- One mistake to avoid when moving to a new marketing technology platform
- Social Media "Lite" for CEOs
- Customer Analytics > Optimizing Your Collection Efforts, Part1
- Time Magazine gets it wrong...so wrong.
- July (9)
- Marketing Accountability
- Do Not Solicit at 36,000 Feet!
- Customer Engagement Agency - the Video
- The Red Thread for Effective Marketing Execution
- Measuring Marketing Efficiency
- Keeping Cable Consumers Happy: Increasingly Difficult - and Important
- Social Media and the Customer Experience
- Thoughts from Neolane Evolution
- "Marketing" versus "Engagement"
- June (9)
- Shooting at Lighthouses - A Lesson for Marketers
- Quaero's InsightIQ Blog: American Business Award Finalist for 2010
- Marketing Key Performance Indicators (KPI’s)
- High Performance Computing in Marketing: Why Should I Care?
- Customer Analytic Visualization
- Marketing Operations 2.0
- Email best practices for B2B lead management
- What is Killing Newspapers? Online Media Business Model and Customer Intelligence
- Evaluating Marketing Performance: The Importance of 'No'
- May (9)
- A Time To Think About Marketing Campaign Quality
- The Top Five Affinium Campaign Features: Optimize Marketing Campaigns
- Getting the Most Out of Your Business Metrics
- A seismic shift from marketing efficiency to marketing effectiveness
- Marketing and Innovation: A Tale of Two Companies
- Customer Intelligence Comes of Age: Reporting from the Forrester Marketing Forum
- Avoiding Analysis Paralysis
- Ahead of the Curve or Pulling Up the Rear?
- The Customer Ecosystem - Part Deux
- April (7)
- Challenges with Mobile Marketing – Part 2
- Data, Data, Everywhere Nor Any Drop of Insight
- Large Analytic Data Environments: What are your options?
- Forrester's Marketing Forum 2010 - are you planning to attend?
- Challenges with Mobile Marketing – Part 1
- The challenges (and occasional perils) of blogging
- Does Marketing Measurement Really Matter?
- March (9)
- Let the Internet Do the Work
- Gaining Momentum with Moments of Truth
- Healthcare Goes Digital
- Customer Experience in the Online World
- Mobile - meet Social Media
- Let's Get Engaged!
- Web Analytics is More Than Just a Reporting Tool
- Building Your Own Online Community
- Time for Marketing to Learn from Software Development
- February (10)
- Shopping vs. Buying Complexity - Draw a Map and Destroy All Obstacles
- Prestige for Sale
- Customer Service and Marketing
- Fewer metrics, greater result
- Google Buzz Case Study - Find Your Own Aha! Moment
- Offer Optimization Framework
- We've Changed!
- Does Brand Marketing still matter?
- Offer Optimization Engine
- Good Intentions, Bad Experience
- January (10)
- Real Time Search and Thoughts on the Marketing Impact
- If the lunatics do run the asylum, how do we control the message?
- Value of Prospect Segmentation Online
- CRM Meets NPO, a love story
- Retention Marketing is the CPR of Attrition.... Too Little Too Late?
- Is technology the gating factor for marketing?
- Honda Uses Facebook to Build an Online Community
- It's All in the Delivery
- Asset Management: the rebranding of AOL & NBC
- Testing, Testing, Testing
-
2009 (158)
- December (6)
- November (9)
- Social Media Is More Than Just PR
- Beware of Click Analytics
- Taking another look at Cross-Collaboration
- Best Practices with Unica Campaign V75 – Should I develop in my production environment?
- Why does Web Analytics give Marketers fits?
- Thoughts on a Recent Neolane Demo
- Missed Opportunity for Customer Engagement
- A bad customer experience has far more impact than a good one ever could.
- Interesting Video on the Social Media Revolution
- October (10)
- Are you building customer loyalty?
- Engagement is the New Loyalty
- 4.0 Tricks of the Trade, Unica V75 – Solving Problems Loading Variable Length Records from Flat Files
- Marketing Roadmaps: Prioitizing Your Marketing Investment
- Care Not Cash
- Marketers prefer Twitter for social media marketing – for now
- Exploring Affinium Campaign Interact 7.5 Concepts - Part 1
- 3.0 Tricks of the Trade in Unica V75 – Printing a Mail List File Even When There are No Records
- CRM - The Devil is in the Details!
- 2.0 Tricks of the Trade in Unica V75 – Copying and Pasting in Unica Campaign
- September (17)
- Analytics > Brainpower versus Software power
- Not exactly a Grand Slamwich
- 1.0 Tricks of the Trade in Unica V75 – How do I dynamically add a date stamp to my list files?
- Kill the CPM, not the messenger
- Marketing's New Frontier - Customer Engagement
- Relax! We're watching for you.
- Email Marketing Best Practices
- The Necessity of Data Profiling
- Thoughts: Adobe's acquistion of Omniture
- Dynamic Sports Advertising
- 4.0 Reusability in Unica 7.5 – Deploying Re-usable Email Document Components
- Evaluating Infrastructure Delivery Options - Post 2 (Cloud)
- Interactive Marketing Spend
- How Financial Services should use web analytics
- Media and Advertising Growth Strategies
- Do you Down Sell?
- Marketing Automation - Are you ready for an upgrade? Part II of II
- August (12)
- A Roadmap for Better Marketing
- Marketing Automation - Are you ready for an upgrade? Part I of II
- Measurement from the Customer's Point of View - Part II
- Evaluating Infrastructure Delivery Options
- Measurement from the Customer's Point of View - Part I
- JetBlue's All-You-Can-Jet Promo- a Boon or Bust?
- At least somebody's making money on Twitter
- Trigger Happy
- Process - the often forgotten component of marketing
- Social media strategy: Three lessons learned
- What's in a name?
- Sustaining Customers via Text Messaging
- July (14)
- Online Analytics in Action - The online data provides the value and it is time to let it loose
- An Eye on Consistent Cross-Channel Messaging
- Good Data Hygiene Never Goes Out of Style
- Digital vs Direct Marketing
- Captain Stan and the art of maintaining a positive customer experience in rough waters
- Top 5 iPhone Apps
- Will hard times spur innovation?
- Monetizing Online Content
- Optimizing Customer Opt-Ins; Preparing for Customer Interaction via SMS/Text Messaging
- Display Advertising Effectiveness - Three Essential Links
- Control Groups in Marketing - Three Essential Links
- Does Customer Value Really Matter?
- Impact of Web Site Performance on Your Bottom Line - Four Essential Links
- Online Analytics in Action - Netflix Competition: Now it’s time to pay up
- June (23)
- Attribution Marketing - Three Essential Links
- Behavioral Targeting - Three Essential Links
- The best thought-out Marketing ROI can still leave you empty handed
- Loyalty Marketing Insights - Three Essential Links
- Data Quality - Three Essential Links
- Retaining HTML Email's Visual Integrity Across Email Clients
- Call Tracking for Marketing Campaigns - Three Essential Links
- Customer Segmentation Best Practices - Three Essential Links
- A Recent Campaign Response Attribution Success Story
- Rewards Program Best Practices - Three Essential Links
- Collaborative Filtering at Netflix - Three Essential Links
- The Paradox of Marketing Automation
- Campaign Attribution - Three Essential Links
- Meaningful Personalization Gone Wrong
- Organizational Alignment - Three Essential Links
- Database Marketing, Charlotte, NC - Localized Marketing - Three Essential Links
- Virtually (in)convenient
- Marketing Data Mart Trends - Three Essential Links
- Social Media Marketing - Three Essential Links
- Digital Marketing Growth in 2009
- Fundamental Truths That Define How Organizations Treat Customers – Law # 4
- Can I get some householding, please?
- Fundamental Truths That Define How Organizations Treat Customers - Law #3
- May (14)
- There must be an app for that!
- How do you know your Website is working if you don't "sell" anything?
- The Real Time Database Complex
- A missed email marketing opportunity
- Selecting Your Marketing Automation Solution - a Two-Part Series (Part II)
- Tales from the Crypt?
- Less Can Be More
- Managing Customers’ Data Quality Expectations
- How Far Will Your Customers Travel?
- Actionable Segmentation in Pharma
- Is Opt-In a Misnomer?
- Do you know your MarketIQ?
- Forrester Marketing Forum 2009: A brief review
- 3.0 Reusability in Unica 7.5 – How to leverage Contact History
- April (12)
- Part 1 of 2 - Selecting Your Marketing Automation Solution
- Everybody loves social networking
- The Real Culprit Behind Unsubscribes – Inbox Clutter isn’t the Problem
- How to Track Down a Moving Target
- Gearing up for Forrester's Marketing Forum
- Smart companies “never allow a crisis to go to waste”
- Future of Direct Mail Marketing
- Fundamental Truths That Define How Organizations Treat Customers - Law #2: People Are Instinctively Self-Centered
- Luring back former customers: Triumph of hope over intelligence?
- Four Tips to Saving on Suppression
- Quick and Dirty Customer Feedback Solutions
- 2.0 Reusability in Unica 7.5 – Really Reusable Offers
- March (20)
- Measure twice, cut once…then execute flawlessly
- Reusability in Unica 7.5
- Privacy & Web Prosperity
- Three links about the future of predictive analytics
- Remember to define your Web analytic strategy
- Three links about the future of decision making
- Mr. Left Hand – Let Me Introduce You to Mr. Right Hand!
- Is email marketing recession proof?
- Twitter?
- Three short links about the future of campaign management
- Silver lining in very dark cloud?
- Buzz from Omniture Summit: Challenges in Online Marketing
- You only need one thing to succeed online
- Read This, Not That!
- Recognizing and Acting on Customer Differences
- A terabyte is not what it used to be...
- "Set it and forget it!"
- Fundamental Truths that Define How Organizations Treat Customers
- A little bit about our own marketing...
- Marketers get social media. They are just sick of the buzz.
- February (17)
- SMS/Texting: The next arrow in pharma/healthcare marketers bag
- US Airways throws in the towel: Score one for the customer!
- Shift from Display to Email and Search Marketing
- Eating Our Own Dog Food
- Web Analytics 101: The Power of the Click
- Customer Experience Management – Challenging to Execute Across All Touchpoints
- Food for Thought
- A New Formula for Loyalty Program Success
- Welcome to the Quaero Web Site!
- Maximizing Value By Understanding Economic Trends
- Timing is everything for email marketing
- The Business of Customer Interactions
- To append or not to append...that is the question.
- Use your space more wisely
- When the going gets tough, innovation suffers
- Attribution in Online Marketing & Advertising
- Customer Intelligence - Still Waiting for the Revolution
- January (4)
- 2008 (2)
From Razvan George Diaconu: Very good article. Useful information. Thanks
From Bill Connolly: Hi Naras, Like you said, the movement yesterday speaks very highly of our country's…
From Naras Eechambadi: Bill, I agree with you. It looks like Congress took a bunch of campaign money from the…