Reddit's database has only two tables

Steve Huffman talks about Reddit’s approach to data storage in a High Scalability post from 2010. I was surprised to learn that they only have two tables in their database.

Lesson: Don’t worry about the schema.
[Reddit] used to spend a lot of time worrying about the database, keeping everthing nice and normalized. You shouldn’t have to worry about the database. Schema updates are very slow when you get bigger. Adding a column to 10 million rows takes locks and doesn’t work. They used replication for backup and for scaling. Schema updates and maintaining replication is a pain. They would have to restart replication and could go a day without backups. Deployments are a pain because you have to orchestrate how new software and new database upgrades happen together.
Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value. There’s a row for every attribute. There’s a row for title, url, author, spam votes, etc. When they add new features they didn’t have to worry about the database anymore. They didn’t have to add new tables for new things or worry about upgrades. Easier for development, deployment, maintenance.
The price is you can’t use cool relational features. There are no joins in the database and you must manually enforce consistency. No joins means it’s really easy to distribute data to different machines. You don’t have to worry about foreign keys are doing joins or how to split the data up. Worked out really well. Worries of using a relational database are a thing of the past.

This fits with a piece I read the other day about how MongoDB has high adoption for small projects because it lets you just start storing things, without worrying about what the schema or indexes need to be. Reddit’s approach lets them easily add more data to existing objects, without the pain of schema updates or database pivots.

.

What is sfsutl, How Should SFSUTL.EXE be Used?

sfsutl.exe can be found in Siebsrvr\bin directory, in siebel versions 6 and older SFSUTL.EXE used to delte the orphan and old versions of attachments.

in Siebel 7 and Siebel 8 a new utility called SFSCLEANUP has been introduced.
sfsutl can be used to verify the integrity between database records and files in the Siebel File System, or to remove files from the Siebel File System that no longer have a corresponding database record.
SFSUTL works in two modes:

1. verify or report

2.move or delete.
1.Verify or Report Mode:
To use the utility in verify or report mode, enter a command string similar to the following:
sfsutl /u sadmin /p sadmin /c siebsrvr_siebel /d dbo /f e:\siebfile /x c:\sfsutl_output.txt
The output from this command could be easily imported into Microsoft utilities like Excel or Access for sorting and manipulation. See below for sample output from this command. There are three key statuses to note:
•Keeping - The file corresponds to a database record.
•Discard - The file does not correspond to a database record, and would be moved or deleted if sfsutl were run in that mode.
•Missing - An expected file does not exist for a database record.
2.Move or Delete Mode:
To use the utility in move or delete mode:
1.Create a separate or new folder (this will be used by sfsutl when moving the discarded files). It is recommended that users do not create this new folder as a subfolder of the existing Siebel File System. For example, if the Siebel File System is in e:\siebfile then create the new folder as e:\siebfilediscards.
2.Enter a command string similar to the following:
sfsutl /u sadmin /p sadmin /c siebsrvr_siebel /d dbo /f e:\siebfile /m e:\siebfilediscards /x c:\sfsutl_output.txt

Siebel Business Services

 

 

• Asynchronous Server Requests
• Data Validation Manager
• EAI Data Transformation Engine
• EAI Dispatch Service
• EAI File Transport
• EAI JMS Transport
• EAI Siebel Adapter
• EAI XML Converter
• EAI XML Queuing Service
• EAI XML Write to File
• EAI XML Read from File
• EAI XSLT Service
• FINS Data Transfer Utilities
• FINS Teller Converter Extensions
• FINS Teller UI Navigation
• Outbound Communications Manager
• PRM ANI Utility Service
• Read CSV File
• Row Set Transformation Toolkit
• SIA BC Utility Service
• SIS OM PC Service
• SIS OM PMT Service
• Shopping Service
• Spell Checker
• SSSE Address Parser (eScript)
• Synchronous Server Requests
• TNT SHM Workflow UI Service
• Transcode Service
• Unified Messaging Service
• Universal Inbox
• Usage Tracking Service
• Web Engine HTTP TXN
• Workflow Process Manager
• Workflow Utilities
• XML Gateway

Tools of the trade

Five years of Siebel EAI development, and I am still learning every day. One thing I quickly realized was that the Siebel Tools and Application system is not like the contemporary Integrated Development Environment (IDE) used in Java and .NET development. Sure, Tools can solve most of your problems, but every Siebel developer will eventually face a problem which has to be solved using some other tool/utility/script which may have to be written from scratch. Today, I thought I’d share what’s in my arsenal of tools.

Siebel Development

1. A good text editor is essential for any development need. For years I have been relying on window’s default notepad utility, which is good enough. But it is seriously lacking in features, and hangs when you try to open huge files. I use Notepad2 , Notepad++ and EditPlus. Each has first class CR/LF support, ANSI to Unicode switching, whitespace and line ending graphics and Mouse Wheel Zooming. EditPlus even has ability search and replace special characters (\n,\t) and works nicely with huuge files.
2. If you work deals with XMLs/XSLs, you will need a good XML/XSLT/WSDL editor. XmlSpy  has been my number one choice, followed by Oxygen. Though I have noticed that there are some minor differences in the XMLSpy XPATH engine between older and newer versions. The XPATH engine in Spy version 6 is similar to Siebel’s own EAI XSLT service. Some of the fine working XSLTs were throwing validation errors in the newer version of Spy, but they worked perfect in Siebel.
3. SOAP UI. This one deserves a special mention. Not only is it free, it has support for secure webservices , proxies and many other protocols. Its so easy to set up a stub/mockservices, and these have saved my life more than once.
4. Fiddler. I first used this way back in college, when I wanted to intercept and see the http messages being sent from my system for a project. It’s a free utility from Microsoft, and comes in handy in diagnosing web service related issues. Its got a scripting system, and can be used as mockservice too.
5. Network tools like PuTTy, WinSCP and FileZilla come here. I also use PuTTY to setup ssh tunnels when I want to route a request via another server.
6. UnixUtils. A unix developer swears by the huge number of commands at his disposal. Sadly, windows does not provide that many commands, and I thank the geeks for porting these awesome Unix commands over to Win32. These allow me to build bat files and scripts to solve recurring issues.
7. Although UnixUtils has windows port of grep and tail commands, the UI of BareGrep and BareTail and much more developer friendly.
8. Oracle SQLDeveloper, TOra and my old favorite, TOAD. You can’t be a Siebel developer without knowing SQL !
9. I’m sure all you Siebel developers have been in this situation: You have list of data you need to query in Siebel :

val1
val2
val3
The searchspec you need to build is : “val OR val2 OR val3” . Well, that’s all that Orit does. Frankly, I don’t know who developed this utility, but it is something developed by a Siebel developer. All it does is simply concatenate column wise data with ‘OR’ in between.

Other Siebel Things I found on the net

Its sad that Siebel does not provide an Out-of-box documentation feature, which can be used during and development. This is just one of the many problems I have faced.Nice thing is others have faced the same problem, and developed their own solutions for the problem.

1. The good folks at the Only Siebel blog have released Excel Macro tools which can pull config data for documentation.
2. Oli has graciously released a lot of his tools for free download. There are script analysers, data loaders and Tracer Tools.
3. Some useful utilities at the website of Sea Marvel Tech Solutions
4. Wait…one more utility for Siebel Documentation here.
5. Here is my own attempt at making an SQL Tracer.
6. Siebel provides dbisqlc to connect and run SQL statements on your localdb. But I find it painful when there are too many columns in the output, I have to scroll to see the data, and export. Sel2XL is a nice excel macro which does the same thing, and it allows excel formatting.
7. Sometimes, the easiest way out is direct SQL into the the database. There are SQLs for siebel issues, EIM Mappings, for UI mappings, and so many other issues.

General Use

1. Window’s default search feature can get irritatingly slow, specially if you need to search across drives and have to run complex search patters. Everything is an awesome ,high speed ,instant search utility for the NTFS file system. You have to try it to see the magic. And did I mention it is free ?
2. Teracopy is the default copy utility on my systems. Sure, the latest multicopy feature on Windows 7 & 8 are cool, but I still feel Teracopy’s interface is better.
3. Free Download Manager. For all those loong downloads.
4. Microsft’s Virtual PC. I use this to setup virtual machines for my Siebel POCs. Its free and no fuss to setup and use. This new version of Windows Virtual PC lets you run Windows XP applications next to your Windows 7 apps for the ultimate in backward-compatibility.
5. I have been using Liberkey suite since they came out some years ago. They have a free utility for all your day to day needs. NirSoft Utilities Collection is also a good choice.
6. Autohotkey, DoItAgain. Ultimate Windows Automation.

View Layout Cache in Siebel Enterprise

This is reblogged from http://ondemand-education.com. Those guys have some cool articles :-)


View Layout Cache in Siebel Enterprise:
This is a subject that comes up often enough in the classroom or in the office, so I thought we would post a short article about it. Before we begin, we need to set the scene. You are navigating through Siebel Call Center or whatever application you use, and you click Help - About View with the Shift key held down. Siebel will kindly tell you how the View Layout you were looking at was brought to you :
Not Cached :

Well, bad news. You are on a dynamic layout view (amongst other things, applets such as Explorer, Hierarchical, Dynamic Drilldowns, Personalization rules can all make the View Layout so dynamic as to be  ”uncacheable”). So the View Layout is never cached.
Server :

Well, so we are getting somewhere. The View Layout was retrieved from the Server and Web Server Cache.
Memory :

Potentially even better. The View Layout came from your Browser Cache. In fact, maybe you just got “Server” as the response, then returned to the same View a few seconds later. The Layout was now in your Browser cache, so you got it from the Browser not your Server. Cool!
Disk :

Your Siebel Object Manager on your Siebel Server has the WebTemplateVersion parameter set and there was no updated View Layout available, so the Persistent Disk Cache version was used. The View Layout you requested was found in the Browser Persistent Disk Cache. This may even have happened between sessions in the Application. Cool!
Digging around in your Browser cache folder will bring all sorts of interesting surprises if this is set up. For example, here is a cached HTML file that was in my Browser just after the previous session :

I guess the point here is to think about the different situations where this may impact us.

• Testing Average Load Times
• Testing Usage Scenarios and making Assumptions
• Choosing View Layout Templates carefully
• Putting Explorer or other dynamic Applets in Views
• Clearing your Browser Cache

There are a number of parameters  both in the CFG side of things and in the User Preferences that can affect the overall behavior, so watch out and take note of the Bookshelf on the subject of Improving Performance.

Siebel 8 Script Libraries

 

Quick question; will the following code snippet work ?

Business Service : BS1, contains only this code

function function1 ()
{
TheApplication().RaiseErrorText("function1  triggered");
}

There is no code in any other event/function of this BS. And now, the attempt is to trigger this BS via the following code:

var bs = TheApplication().GetService("BS1");
bs.function1();

Now there is something wrong about the second code snippet, right ? This is not the usual way to invoke a Business Service Method.  The practise is to use the InvokeMethod command, passing property sets for input and output.  But here is the output of running these in Siebel 8

 

This is an example of Script Libraries feature from Siebel 8 onwards. Developers can write multiple functions in business services, and then these functions get exposed , and the functions can be invoked directly as you would do on C/C++/Java. There is no need of adding code in  Service_PreInvokeMethod  event to expose the functions.

There are limitations though, such a business service’s functions can be invoked only via scripting. They cannot be used in WFs or BRPs. But if your functionality calls for lots of scripting, this feature surely comes in handy.

 

The ever friendly Oli has been posting some really tricky pieces of code for his code challenges. Head over there to learn scripting mistakes that creep up in code.

Happy Scripting !

View Refresh when clicking New Record in a view with Dynamic Toggle

 

Dynamic toggle applets were probably the first piece of automation a Siebel developer gets to work on; switch the applet depending on some field value. No scripting, nothing at the BusComp level, just Applet toggles. But issues crop in when the logged in user tries to create new record on the applet. Sometimes the view jumps or refreshes, specially when there are lots of applets stacked in the view. The user has to manually scroll down back to this target applet.

Oracle has a work around for this ‘defect’ documented here [ID 541100.1] , which involves loads of scripting at the applet and BusComp level. But I tried to come up with something with fewer lines of code.

Resulting solution: add the following code in the WebApplet_PreInvokeMethod section of the base applet as well as its toggle applets:

 

if (MethodName == "NewRecord")
{
this.BusComp().NewRecord(NewBefore);
return (CancelOperation);
}

 

Yep, I know, the code does not make sense at all. But for some reason, it works ! The view does not jump and the new record gets created right there in the applet. At at just 3 lines of code, it beats oracles long and elaborate code version.

Desk.Com- Service Cloud for SMBs

On Jan 31st Salesforce unveiled its customer service application for small & medium business enterprises.
It is called Desk.com & is based on Salesforce's acquisition of Assistly.
Desk.com is a cloud based offering for SMBs to support their customers.

Key Features -

• Build with keeping the Salesforce Social theme in the core, Desk.com allows the companies to support their customers over the major social channels like Facebook & twitter.


• Integration with Facebook & Twitter is the standard feature of the product and it takes few clicks to link organization's FB & Twitter accounts with Desk.com.


• Any or all the Tweets & FB posts on the linked accounts can be created & tracked as cases in Desk.com.


• Desk.com also supports all the traditional customer support channels like Phone, Chat & Email.


• Organizations can create a knowledge bank which can be made available to customers via their websites. This knowledge bank can act as the 1st step for the customers to resolve their issues.

Salesforce has also launched Desk.com for Mobile platforms. Desk.com for mobile is a HTML5 based application which supports all the major mobile platforms. Agents can respond to the customer while on move. All the major case management functionalities like sending responses, changing case priority, escalating the case etc. are available via Desk.com Mobile.

Pricing - 1st user license is completely free, create your account & start using it. After that it's US$ 49 per agent per month for unlimited usage. For part time support agents there is a flexible pricing option available which is US$ 1 per hour per user.

In this fast changing digital world where people spend a huge chunk of their time on internet nobody can ignore or deny the power of Social media.
But if Salesforce is targeting SMB's for this product then I am not sure how much the social part can be utilized by these organizations. Social interactions require a dedicated team to respond to Social media and if the responses are not handled by the experts then it can boomerang on the company and can have lasting negative impacts.

SMBs generally have very small customer service teams and they would like to respond to the actual customers/prospects rather than people posting random thoughts and queries on social media websites.
So I believe initially Desk.com will primarily be used for its traditional channel support & when the company grows in size and has enough support staff then they can start using the Social part of Desk.com.


Reference -http://www.youtube.com/watch?v=gFEbcDojo1A&feature=related

eScript–Nested ‘with’ has problems in 8.1.1.5

Recently we had the friendly guys from Oracle come over and review our current project. Over the years, we have had review comments coming from such reviews and now know what to expect. But this time, there was a new entry in the document.

Siebel eScript developers and basically anyone who has worked on ECMA style languages must have used the ‘with’ statement. The with statement assigns a default object to a statement block, so you need to use the object name with its properties and methods. Its makes coding easier when you need to do multiple actions on the same object. But nesting with statements , it seems, is not a good idea if you are planning to upgrade to version 8.1.1.5 which came out last year.

The With statement structure indicates that all methods within its block will be based primarily on the indicated object. When With blocks are nested, it is not immediately obvious which object’s method will be invoked. The code execution may not do what the developer intended.

with(firstbc)

{

ClearToQuery()

ExecuteQuery();

with(secondbc)

{

ClearToQuery()

ExecuteQuery();

}

}

If the script remains unchanged prior to upgrading to 8.1.1.5, there is a known defect where runtime errors will occur. Although this is currently considered a defect and intended to be corrected, nested With blocks are not a recommended scripting practice. All of the methods invoked in the second With block would also work on the object in the first With block. In this script, the developer was actually done using the firstbc object prior to starting the nested With, but simply chose not to close the block.
Now oracle says that :It is not recommended to nest With blocks. The first With block should be close prior to initiating a new With block or the object variables should always be used.
Now we have used countless nested with statements it handle complex business logic, and have never faced an issue. But we are now rewriting the code eliminating nested withs and using the complete object names. This is because we do have plans to upgrade some time in future, and its best to steer clear of rework then.

with(firstbc)

{

ClearToQuery()

ExecuteQuery();

secondbc.ClearToQuery()

secondbc.ExecuteQuery();

}

Update: Oracle SRs are here and here

New Year, first post.

 

Happy New year ,everyone. Yeah, I know, this post is long overdue.  I changed jobs some time last year, and the work at the new place is not exactly what I expected.  Crazy deadlines, unrealistic requirements, last minute changes…the works.

But I did get to learn more about this whole CRM world.. And here’s hoping I find more time to share more of what I learn.

I began my career on Siebel 5 years ago, and it has been my bread and butter. The tried and tested On-Premise mode of CRM installation has always been popular with the blue chip and Fortune 500 clients I had the opportunity to work for.  Although cloud based applications are gaining foothold, most of my employer’s clients steered away from sharing mission critical data on the web. They seem to feel more comfortable maintaining and storing their customers data in company’s storage rooms. A lot of them have have simply said no to SalesForce CRM because they don”t get to secure their customer’s data. But all that is changing.

SalesForce.com has understood this customer concern, and the have decided to do something about it. This year, they will introduce a new feature called Data Residency Option or DRO. Simply put, DRO will enable On Premise storage of mission critical data on Cloud.com servers, which can be setup inside client office locations.

DRO will be a part of database.com - a cloud database Salesforce made generally available. It gives an option to the customers of Salesforce to store their mission critical data at their own location and hence keeping complete control of the inward and outward flow of the data across the customer firewall.

The technique developed by Navajo, also called Virtual Private Saas, provides the cloud vendor, Salesforce.com in this case, a key that enables it to translate the encrypted data as it passes through its cloud application. The data is then re-encrypted as it leaves the cloud vendor's solution and returns to the customer's data source. The corporate data is unreadable on cloud provider's servers during this entire operation. VPS is available both as a cloud service, as well as an appliance sitting on the customer's local or Wide area network. With VPS, the customer is solely responsible for its data security as it will hold all the encryption keys.

The flip side to using such a technique would be the security of the encryption and decryption keys used for the process. It is highly critical to properly manage the keys as once the key is lost, the encrypted data can no longer be accessed. Hence, this calls for robust key management to avoid any such eventuality.

But, barring the above, In my view, this technique will overcome the most important impediment to cloud adoption and will be a foundation of technological acceptance as it addresses the key customer fear i.e. about potential data threats in the cloud.

Coming to the acquisition, Navajo systems, founded in 2009 was one of the existing encryption service providers for Salesforce. Salesforce decision to acquire Navajo hence made a lot of sense when other cloud based CRM tools such as Sugar CRM already has possible options for deployment on public clouds (Amazon EC2, Rackspace etc.), private clouds such as VMWare and also on-site behind customer firewalls.

According to a recent report from IBIS World, one of the world's largest independent publishers of U.S. industry research, CRM industry today stands at 60% on-premise deployments and 40% cloud based solutions (1). For customers who are looking for new purchases or upgrade of their legacy applications, DRO might just be the key decision influencer. Let's wait and watch!!

References:

(1) http://www.destinationcrm.com/Articles/Columns-Departments/Insight/Are-CRMs-Worst-Years-Behind-It-79254.aspx